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MEMORANDUM  FOR  ACTING  UNDER  SECRETARY  OF  DEFENSE  (PERSONNEL  AND 

READINESS) 

SUBJECT:  Review  of  the  Scientific  Evidence  of  Using  Population  Normative  Values  for  Post- 
Concussive  Computerized  Neurocognitive  Assessments 

The  Defense  Health  Board  (DHB)  is  pleased  to  submit  its  report  summarizing  the  findings  and 
recommendations  from  its  independent  review  of  the  Scientific  Evidence  of  Using  Population 
Normative  Values  for  Post-Concussive  Computerized  Neurocognitive  Assessments. 

On  July  25,  2014,  the  Undersecretary  for  Personnel  and  Readiness  (USD(P&R))  tasked  the  DHB 
to  review  the  scientific  evidence  for  using  normative  values  versus  individual  baselines  for  post- 
concussive  computerized  neurocognitive  assessments.  In  response,  the  DHB  assigned  the 
Neurological/Behavioral  Health  Subcommittee  to  examine  the  state  of  the  science  on 
computerized  neurocognitive  testing  and  develop  proposed  recommendations  for  the  Board’s 
consideration. 


The  Subcommittee  conducted  an  in-depth  literature  review,  received  briefings  from  subject 
matter  experts,  and  conducted  panel  discussions  with  leaders  in  the  field  of  mild  traumatic  brain 
injury  (mTBI)  and  neurocognitive  testing.  Following  public  deliberation  of  the  findings  and 
recommendations,  the  attached  report  was  finalized. 


On  behalf  of  the  DHB,  I  appreciate  the  opportunity  to  provide  the  Department  with  this 
independent  review  and  hope  that  it  provides  useful  information  to  further  advance  health  care 
for  our  military  members. 

President,  Defense  Health  Board 


Attachment: 
As  stated 


cc: 

ASD(HA) 
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The  recent  conflicts  in  Iraq  and  Afghanistan  have  raised  concerns  about  the  “invisible  wounds 
of  war”1  endured  by  Service  members  following  deployments.  Mild  traumatic  brain  injury 
(mTBI),  also  referred  to  as  concussion,  is  among  those  injuries  and  remains  difficult  to  evaluate 
and  manage.  The  long-term  effects  of  mTBI  can  leave  Service  members  with  insomnia,  anxiety, 
emotional  distress,  and  impaired  cognitive  functioning.  As  of  March  2015,  there  have  been  more 
than  320,344  diagnosed  cases  of  traumatic  brain  injury  (TBI)  (in  theater  and  in  garrison)  in  the 
U.S.  military  since  2000.“  In  response  to  the  large  number  of  TBIs,  Congress  included  a 
provision  in  the  2008  National  Defense  Authorization  Act  (NDAA)  stating  that  the  “Secretary  of 
Defense  shall  develop  and  implement  a  comprehensive  policy  on  consistent  neurological 
cognitive  assessments  of  members  of  the  Armed  Forces  before  and  after  deployment.”3  The 
Department  of  Defense  (DoD)  implemented  this  requirement  through  a  DoD  Instruction  that 
requires  all  Service  members  who  deploy  to  undergo  baseline  neurocognitive  testing  within  12 
months  prior  to  deployment.4  The  Automated  Neuropsychological  Assessment  Metrics  4 
(ANAM4),  a  computerized  neurocognitive  assessment  tool  (NCAT)  developed  by  the  U.S.  Army 
to  assess  the  cognitive  function  of  Service  members,  was  selected  to  support  this  effort.  The 
individual  baseline  measurements  (baselines)  obtained  using  ANAM4  were  intended  to  provide  a 
basis  for  comparison  and  evaluation  in  the  event  that  a  Service  member  experienced  an  mTBI 
during  his  or  her  deployment.  As  of  September  2015,  more  than  1.8  million  baselines  have  been 
completed.  An  initial  normative  database  (a  reference  set  of  baseline  tests  from  the  population  of 
interest)  was  created  in  2008  with  107,000  tests  stratified  by  age  and  sex.  There  are  efforts 
currently  underway  to  expand  the  normative  database,  and  to  date  approximately  1 . 1  million 
ANAM4  test  results  have  been  selected  for  analysis  and  inclusion  in  the  new  database.5 

Given  the  size  of  the  current  ANAM4  database  and  recent  publications  indicating  that  using  a 
normative  data  set  for  comparison  may  be  as  effective  as  relying  on  individual  baselines  for 
comparison  in  assessing  cognitive  function  following  mTBI,6’7  questions  have  been  raised  about 
the  utility  of  continuing  to  collect  pre-deployment  baselines.  To  address  these  questions,  on  July 
25,  2014,  the  Undersecretary  for  Personnel  and  Readiness  (USD(P&R))  tasked  the  Defense 
Health  Board  (DHB)  to  review  the  scientific  evidence  for  using  population  normative  values  for 

o 

post-concussive  computerized  neurocognitive  assessments.  The  DHB  Subcommittee  on 
Neurological/Behavioral  Health  examined  the  state-of-the-science  on  neurocognitive 
assessments  and  provided  recommendations  regarding  the  need  for  continued  baseline  testing  of 
individuals,  the  adequacy  of  the  current  normative  dataset,  the  utility  of  expanding  the  use  of 
neurocognitive  testing  beyond  the  deployment  cycle,  areas  for  future  research,  and  the  cost- 
benefit  tradeoffs  of  performing  baseline  testing. 

Automated  Neuropsychological  Assessment  Metrics  4  History  and  Use 

The  ANAM4  is  designed  to  assess  a  Service  member’s  cognitive  function  after  he  or  she  has  had 
an  mTBI.  NCATs  are  relatively  new  in  the  field  of  mTBI  testing.  The  first  computerized 
NCATs  were  developed  in  the  late  1980s  and  were  first  used  in  the  diagnosis  and  management  of 
mTBI  during  the  latel990s.  Their  use  has  gained  popularity  in  both  professional  and  amateur 
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sports  organizations  in  addition  to  their  use  in  pre-deployment  baseline 
testing  of  Service  members.9 

DoD  began  developing  the  ANAM  test  battery  in  1984  to  assess  the  impact  of  various  drugs, 
medications,  and  nerve  agents  on  the  cognitive  performance  of  Service  members.10  The 
resulting  test  battery  provided  a  basis  for  understanding  effects  of  medication  and  other 
environmental  factors  on  the  brain,  but  it  was  not  standardized  or  well  suited  for  the  assessment 
of  long-term  conditions.  The  test  battery  has  since  been  revised  to  focus  its  assessment  on  the 
detection  and  evaluation  of  neurocognitive  deficits  that  result  from  mTBI.10  In  theater,  clinical 
guidelines  suggest  that  the  test  battery  be  administered  if  symptoms  of  concussion  are  present  24 
hours  post-injury.  This  first  administration  should  be  completed  between  24  to  72  hours  post¬ 
injury.1 1  The  test  may  be  re-administered  on  a  regular  basis  as  symptoms  resolve  to  assess  for 
persistent  neurocognitive  deficits  and  to  inform  the  retum-to-duty  decision-making  process.  A 
Service  member’s  in-theater  post-injury  ANAM4  scores  may  be  compared  to  his  or  her 
individual  baseline,  if  available,  or  to  the  normative  database  to  determine  whether  cognitive 
impairment  is  present.12 

ANAM4  plays  a  specific  role  in  the  evaluation  and  management  of  mTBI.  While  valuable  as  an 
indicator  of  cognitive  function,  it  is  important  to  note  that  the  test  is  only  one  component  of  a 
comprehensive  clinical  assessment  and  that  test  results  should  be  interpreted  by  a  trained 
neuropsychologist  in  that  context.13'14 

Sensitivity,  Specificity,  and  Reliability  of  Computerized  Neurocognitive 
Testing 

Current  DoD  policy  mandates  that  each  Service  member  complete  ANAM4  baseline  testing 
within  12  months  prior  to  deployment.  However,  recent  budget  constraints  in  combination  with 
new  research  suggesting  that  normative  data  may  be  sufficient  to  identify  impairment  in 
neurocognitive  function  post-mTBI  has  prompted  a  review  of  the  policy  and  its  associated  costs. 

Comparing  the  relative  effectiveness  of  individual  baseline  ANAM4  results  and  normative  data 
in  post-injury  assessments  is  complex.  Most  experts  agree  that,  logically,  a  valid  high-quality 
individual  baseline  would  provide  the  most  accurate  data  for  clinicians  to  make  a  well-informed 
return-to-duty  decision.  However,  recent  research  indicates  that  collecting  an  accurate  baseline 
is  challenging.111516  Other  elements  that  must  be  considered  in  comparing  these  two  approaches 
are  the  value  that  baseline  testing  provides  for  individuals  who  ordinarily  perform  significantly 
above  or  below  the  norm;  challenges  in  appropriately  interpreting  ANAM4  test  results;  and  the 
size  and  stratification  of  the  existing  military  ANAM4  normative  database. 

Finding  1.1  :  Current  evidence  is  inconclusive  on  whether  using  individual  baseline 
computerized  neurocognitive  assessment  test  results  is  more  advantageous,  on  a 
population  level,  than  using  an  optimally  stratified  normative  dataset  in  evaluating  and 
managing  mild  traumatic  brain  injury,  including  assessing  return  to  duty  decision  making 
and  prognosis. 

Finding  1.2  :  The  current  ANAM4  military  normative  dataset  is  stratified  solely  by  age 
and  sex  and  does  not  accurately  estimate  baseline  neurocognitive  function  in  individuals 
who  score  at  the  high  or  low  ends  of  the  scale.  Efforts  are  currently  underway  to  expand 
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and  further  stratify  this  dataset  to  improve  accuracy  in  estimating 
cognitive  deficits  related  to  mild  traumatic  brain  injury. 

Recommendation  1:  DoD  should  continue  to  analyze  existing  ANAM4  data  to 
determine  whether  an  optimally  stratified  normative  dataset  can  be  developed  that  is 
capable  of  accurately  estimating  baseline  neurocognitive  function,  including  for 
individuals  who  score  at  the  high  or  low  ends  of  the  scale.  Specific  stratification 
variables  to  consider  adding  include  education,  rank,  standardized  test  scores, 
race/ethnicity,  and  environmental  factors  such  as  socioeconomic  status. 

Finding  2:  It  is  not  apparent  that  an  adequate  overall  assessment  of  the  utility  of  the 
current  ANAM4  testing  program  in  evaluating  and  managing  mild  traumatic  brain  injury 
has  been  accomplished.  Moreover,  it  is  not  apparent  that  clinical  evaluation  and 
disposition  data  have  been  centrally  compiled  to  accomplish  this  analysis. 

Recommendation  2:  DoD  should  analyze  current  clinical  data  from  mild  traumatic 
brain  injury  evaluations  to  determine  in  what  proportion  of  cases  ANAM4  testing 
provided  information  of  clinical  value  that  was  a  contributing  factor  in  overall 
management  and  disposition  of  the  patient.  As  part  of  this  analysis,  the  value  of  having 
an  individual  baseline  for  comparison  should  be  assessed. 

Finding  3:  Current  evidence  is  inconclusive  regarding  the  value  of  routine  pre¬ 
deployment  baseline  ANAM4  testing. 

Recommendation  3:  DoD  should  request  that  legislation  requiring  routine  pre¬ 
deployment  neurological  cognitive  assessments  be  rescinded  to  allow  discontinuation  of 
routine  pre-deployment  ANAM4  baseline  testing.  DoD  should  instead  conduct 
prospective  research  using  neurocognitive  or  other  assessment  tools  to  evaluate  their 
usefulness  in  the  management  of  mild  traumatic  brain  injury  and  return-to-duty  decision¬ 
making. 

NCATs  were  designed  to  track  recovery  from  traumatic  brain  injury.  They  were  not  specifically 
designed  to  assess  prognosis,  and  studies  have  not  shown  that  NCAT  test  results,  including  those 
of  ANAM4,  are  significantly  predictive  of  prognosis  following  mTBI.  However,  there  is  some 
evidence  that  better  performance  on  simple  reaction  time  subtests  may  be  indicative  of  shorter 
recovery  time.  Research  on  clinical  cases  indicates  that  loss  of  consciousness  and  the  length  of 
time  that  an  individual  is  unconscious  following  a  TBI  are  correlated  with  increased  recovery 


Finding  4:  ANAM4  testing  has  not  yet  been  shown  to  consistently  or  significantly 
contribute  to  assessing  long-term  prognosis  after  a  mild  traumatic  brain  injury. 

Recommendation  4:  DoD  should  analyze  post-injury  test  scores  and  long-term  data 
documenting  the  recovery  of  Service  members  after  a  mild  traumatic  brain  injury  to 
determine  if  post- injury  ANAM4  scores  consistently  or  significantly  contribute  to  the 
assessment  of  prognosis. 


Executive  Summary 


ES-3 


Defense  Health  Board 


Automated  Neuropsychological  Assessment  Metrics  4 
Normative  Database  and  Data  Repository 

The  U.S.  military  has  collected  more  than  1.8  million  baseline  ANAM4  test  results,  which  have 
been  incorporated  into  a  searchable  database.  Test  scores  for  individual  Service  members  are 
available  24/7  so  that  clinicians  may  conduct  post-injury  evaluations  at  any  time,  whether  in 
theater  or  in  garrison.  To  date,  1.1  million  test  scores  have  been  deemed  suitable  for  analysis  and 
incorporation  into  an  updated  normative  database  that  will  be  further  stratified  to  improve 
statistical  accuracy  in  identifying  changes  in  cognitive  function  when  evaluating  mTBI  with 
post-injury  ANAM4  test  scores.5  Research  indicates  that  demographic  factors  such  as  education, 
rank,  ethnicity,  and  handedness  may  affect  performance  on  neurocognitive  testing  and  that  the 
creation  of  norms  that  reflect  these  descriptors  will  improve  the  accuracy  of  a  normative 
database.19'20  Since  the  characteristics  of  the  military  population  will  likely  change  over  time, 
efforts  should  be  made  to  collect  and  update  the  ANAM4  baselines  to  reflect  the  current 
demographic  distribution  of  the  military  population  to  maintain  a  current  normative  database. 

Automated  Neuropsychological  Assessment  Metrics  (ANAM)  Program 
Costs 

As  a  result  of  the  2008  NDAA  requirement  for  pre-deployment  baselines,  the  magnitude  and 
costs  associated  with  the  ANAM  program  increased.  Licensing  and  contracting  expenses,  as 
well  as  costs  for  computers  and  other  infrastructure,  increased  to  accommodate  the  large  number 
of  tests  performed  on  deploying  Service  members.  The  overall  cost  continued  to  increase 
through  2011  and  then  began  to  decrease  starting  in  2012,  correlating  with  a  decrease  in  the 

9  1 

number  of  troops  deployed.  DoD  currently  pays  an  annual  licensing  fee  to  use  the  ANAM4 
tool,  as  well  as  a  contracting  fee  that  covers  testing  administration.  These  costs  total 
approximately  $5  million  per  year.5  DoD  also  uses  the  Immediate  Post-Concussion  Assessment 
and  Cognitive  Testing  (ImPACT)  tool  to  evaluate  Service  members  who  are  part  of  the  U.S. 
Military  Special  Operations  units,  with  an  approximate  cost  of  $2  per  baseline  test  and  $8  per 
post-injury  test.  ImPACT  and  other  NCATs  have  demonstrated  similar  accuracy  and 
effectiveness  to  ANAM4  in  evaluating  mTBI. 

Baselines  comprise  the  vast  majority  of  total  ANAM4  tests  administered.  Post-deployment, 
clinical,  injury,  and  recovery  assessments  comprise  only  4.6  percent  of  the  total,  and  tests 
conducted  for  unspecified  reasons  comprise  1 .7  percent,  indicating  that  baselines  are  not 
accessed  for  the  majority  of  the  population  that  has  undergone  testing.5  Additionally,  data 
provided  by  the  Defense  and  Veteran’s  Brain  Injury  Center  show  that  80  percent  of  concussions 
occur  in  garrison,  where  baseline  testing  is  not  required,  and  other  neuropsychological 
assessments  may  be  used  for  evaluation.  In  light  of  the  current  fiscal  environment,  as  well  as  the 
decrease  in  number  of  deployments,  questions  have  been  raised  regarding  the  cost-effectiveness 
of  the  current  ANAM  program  and  the  continued  need  for  baseline  testing. 


Finding  5:  Currently,  there  is  insufficient  evidence  to  support  expanding  baseline  testing 
outside  of  the  deployment  cycle. 
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Recommendation  5:  DoD  should  defer  a  decision  to  expand 
ANAM4  baseline  testing  outside  of  the  deployment  cycle  until  additional  research 
demonstrates  that  baseline  testing  improves  the  evaluation  and  management  of  mild 
traumatic  brain  injury. 

Finding  6:  DoD  is  currently  using  at  least  two  NCATs  (ANAM  and  ImPACT)  to  assess 
neurocognitive  function.  There  are  significant  differences  in  the  pricing  structure  for 
these  tools  and  studies  have  not  shown  either  tool  to  have  a  distinct  overall  advantage  in 
contributing  to  the  management  of  mild  traumatic  brain  injury. 

Recommendation  6:  DoD  should  conduct  a  competitive  bidding  process  to  select  the 
most  cost-effective  approach  to  meet  requirements  for  all  non- specialized  neurocognitive 
assessments  for  the  management  of  mild  traumatic  brain  injury. 

Areas  for  Future  Research 

Recent  publications  outline  the  need  for  additional  research  on  the  development  and  appropriate 
use  of  NCATs13’16'24'25.  These  tools  have  become  widely  used  in  the  assessment  and 
management  of  mTBI;14  however,  challenges  remain  in  improving  their  sensitivity,  accuracy, 
and  the  methodology  for  interpretation  of  test  results.  Areas  for  future  research  include  cost- 
effectiveness  of  baseline  testing,  accuracy  of  using  baseline  versus  normative  values  in  assessing 
mTBI,  the  development  of  multimodal  assessments  methods,  validity  and  accuracy  of  other 
computerized  neurocognitive  assessment  tools,  test  reliability  of  ANAM4  and  its  individual 
subtests,  symptomatology  of  and  recovery  from  mTBI,  ANAM4  test  score  interpretation 
methodology,  and  the  effect  of  testing  environment  on  NCAT  scores.  Of  special  interest  to  DoD 
are  new  assessments  utilizing  brain  imaging,  postural  stability,  vision  testing,  biomarkers,  and 
other  physiological  evaluations.  There  is  increasing  evidence  that  these  multimodal  assessments, 
in  addition  to  neurocognitive  testing,  may  provide  a  more  effective  approach  to  the  evaluation 
and  management  of  mTBI.14’26,27 

Finding  7:  There  is  evidence  from  academic  research  and  from  DoD  that  multimodal 
approaches  including  imaging,  use  of  biomarkers,  and  physical  diagnostic  techniques 
may  be  more  effective  than  NCATs  alone  in  evaluating  and  managing  mild  traumatic 
brain  injury. 

Recommendation  7:  DoD  should  sustain  and  advance  research  to  determine  if  a 
multimodal  approach  can  be  developed  that  is  cost-effective  and  superior  to  NCAT 
testing  alone  in  assessing  and  managing  mild  traumatic  brain  injury. 


Finding  8:  Individuals  with  persistent  symptoms  following  mild  traumatic  brain  injury 
are  often  found  to  have  comorbidities  such  as  post-traumatic  stress,  depression,  sleep 
disturbances,  pain,  and  anxiety.  There  is  insufficient  research  on  the  impact  of  these 
comorbidities  on  ANAM4  test  performance. 

Recommendation  8:  DoD  should  conduct  additional  research  to  determine  the  effects  of 
comorbidities  on  ANAM4  test  performance. 
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Finding  9:  DoD  has  collected  more  than  1.8  million  baseline, 
clinical,  and  other  ANAM4  test  results. 

Recommendation  9:  DoD  should  make  a  deidentified  version  of  these  data  available  to 
civilian  researchers  to  leverage  those  resources  in  accomplishing  additional  analyses. 


Finding  10:  DoD  currently  administers  ANAM4  baseline  tests  in  group  settings; 
however,  research  indicates  that  administering  computerized  neurocognitive  tests  in  a 
group  setting  may  affect  an  individual’s  test  score." 

Recommendation  10:  DoD  should  determine  whether  and  to  what  extent  the  group 
testing  environment  affects  ANAM4  baseline  tests  scores  as  compared  to  individual 
ANAM4  testing. 
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1.  Background  and  Introduction 

The  recent  conflicts  in  Afghanistan  and  Iraq  have  brought  increased  attention  to  the  “invisible 
wounds”  of  war1,  such  as  traumatic  brain  injury  (TBI)  and  post-traumatic  stress  disorder 
(PTSD).  ’  These  invisible  wounds  can  have  far-reaching  impact,  affecting  not  only  Service 
members,  but  their  families  as  well.  The  long-term  effects  of  PTSD  and  TBI  include  insomnia, 
anxiety,  emotional  distress,  or  impaired  cognitive  functioning.1  These  symptoms  also  affect 
families,  as  spouses  and  children  must  leam  to  adjust  their  behavior  and  lifestyle  to  protect  and 

•3  1 

care  for  the  Service  member,  often  causing  secondary  anxiety.  PTSD  and  TBI  can  be  isolating, 
leaving  Service  members  and  their  families  disconnected  from  their  regular  social  activities 
because  of  the  stress  of  coping  with  unpredictable  situations.  These  wounds  manifest  differently 
depending  on  the  person,  and  the  medical  community  still  struggles  to  diagnose  and  provide 
effective  treatment  for  these  conditions.  This  report  focuses  on  mild  TBI  (mTBI),  or  concussion, 
and  aims  to  identify  the  most  effective,  fiscally  responsible  approach  to  using  computerized 
neurocognitive  assessment  testing  in  the  management  of  mTBI. 

Since  the  start  of  Operation  ENDURING  FREEDOM  (OEF)  and  Operation  IRAQI  FREEDOM 
(OIF)  the  number  of  diagnosed  TBIs  has  increased  as  awareness  and  medical  technology  have 
improved.32  As  of  March  2015,  there  have  been  more  than  320,344  diagnosed  cases  (in  theater 
and  in  garrison)  of  TBI  in  the  U.S.  military  since  2000,  and  it  is  assumed  that  the  actual  number 
of  TBIs  is  greater  due  to  a  lack  of  reporting  and  challenges  in  identifying  subtle  symptoms  that 

9 

occur  with  mTBI.  Because  of  the  increasing  number  of  diagnoses  and  increased  focus  on  these 
injuries,  the  Institute  of  Medicine’s  Committee  on  the  Assessment  of  the  Readjustment  Needs  of 
Military  Personnel,  Veterans,  and  Their  Families  recognized  them  as  the  signature  wounds  of 
OEF/OIF.32  Indeed,  the  high  incidence  of  brain  injury  among  Service  members,  both  deployed 
and  in  garrison,  has  underscored  the  need  for  improved  diagnosis,  treatment,  and  prevention  of 
TBI.1 

Mild  TBI,  commonly  referred  to  as  concussion,  is  defined  as  “a  traumatically  induced  structural 
or  physiological  disruption  of  brain  function  as  the  result  of  an  external  force  that  is  indicated  by 
the  onset  or  worsening  of  at  least  one  of  the  following  clinical  signs  immediately  following  the 
event:  loss  of  consciousness  lasting  less  than  30  minutes;  alteration  of  consciousness  or  mental 
state  lasting  up  to  24  hours;  posttraumatic  amnesia  up  to  24  hours;  or  Glasgow  Coma  Scale  (best 
available  score  during  the  first  24  hours)  of  13-15.”  Patients  frequently  report  the  sensation  of 
“seeing  stars”  or  having  had  their  “bell  rung.”2  Following  the  acute  phase  post-injury,  symptoms 
are  less  distinct  and  are  often  attributed  to  other  causes.  Headache,  dizziness,  sleep  disturbances, 
poor  coordination,  change  in  mood,  and  lack  of  focus  frequently  manifest;  however,  individuals 
may  not  attribute  those  symptoms  to  their  injury  or  choose  not  to  report  their  symptoms  to  avoid 
removal  from  duty.3’  Even  if  symptoms  appear  mild,  the  brain  is  vulnerable  to  repeat  trauma 
following  an  mTBI.  Appropriate  management  is  vital  to  ensure  that  Service  members  who 
experience  mTBI  do  not  place  themselves,  others,  or  their  mission  at  risk.34 

Recognizing  the  need  for  improved  screening  and  diagnosis  of  mTBI,  the  U.S.  Army  began 
research  on  computerized  neurocognitive  assessments  in  consultation  with  military  and  civilian 
experts  in  2001. 10  Efforts  focused  on  the  Automated  Neuropsychological  Assessment  Metrics 
(ANAM),  a  test  initially  developed  in  1984  to  assess  the  impact  of  various  drugs  on  the  cognitive 
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function  of  members  of  the  U.S.  military.10  The  resulting  computerized  assessment  consisted  of 
a  battery  of  tests  intended  to  evaluate  cognitive  processing  speed,  resistance  to  interference,  and 
working  memory.  ANAM  testing  was  not  broadly  implemented  in  the  military  until  2008,  when, 
amidst  growing  concern  over  the  lasting  effects  of  TBI,  Congress  passed  a  bill  requiring  all 
Service  members  who  deploy  to  undergo  baseline  neurocognitive  testing  within  12  months  prior 
to  their  deployment.3  The  intent  was  to  ensure  that  all  Service  members  would  have  a  baseline 
that  clinicians  could  compare  with  post- injury  or  post-deployment  ANAM  scores  to  determine 
whether  cognitive  impairment  was  present. 

The  ANAM,  currently  in  its  fourth  iteration  (ANAM4),  consists  of  22  modules  that  are  intended 
to  detect  changes  in  cognitive  function  that  occur  because  of  injury,  illness,  or  exposure.  A  list 
of  the  modules  is  included  in  Appendix  C.  Since  its  implementation  in  2008,  the  ANAM  has 
evolved  to  incorporate  improved  measures  for  effort  and  validity  and  further  research  has  been 
conducted  to  assess  its  reliability,  sensitivity,  and  specificity.  ’  ’  ’  '  Regulations  and 
guidelines  have  been  developed  to  aid  in  its  effective  application.  A  2013  Department  of 
Defense  Instruction  (DoDI)  states  that  “Neurocognitive  assessment  tools  will  be  used  in  a 
screening  capacity  to  detect  cognitive  changes  as  part  of  a  clinical  evaluation  and  will  not  be 
used  as  a  standalone  diagnostic  tool.”4  Currently,  the  ANAM  must  be  administered  within  12 
months  prior  to  deployment,  but  it  is  not  routinely  administered  following  a  deployment  unless  a 
Service  member’s  post-deployment  questionnaire  indicates  that  they  may  have  experienced  a 
TBI  while  deployed.  The  Defense  Centers  of  Excellence  for  Traumatic  Brain  Injury  and 
Psychological  Health  has  also  developed  guidelines  for  the  assessment  and  management  of  acute 
mTBI.  If  a  post-injury  or  post-deployment  ANAM  is  required,  the  clinician  may  compare  the 
results  of  the  test  to  the  Service  member’s  baseline  to  aid  in  their  overall  evaluation.  This  post¬ 
injury  test  is  a  tool  used  in  conjunction  with  other  assessments  and  clinical  judgment  to 
determine  whether  a  Service  member  may  return  to  duty. 

The  current  standard  for  evaluating  mTBI  is  comparing  a  Service  member’s  post-injury  ANAM 
scores  to  his  or  her  baseline  score.  Ideally,  this  provides  a  reliable  point  of  reference  for  the 
evaluating  medical  professional  to  determine  whether  the  Service  member  shows  any  decline  in 
cognitive  function  that  might  be  related  to  his  or  her  mTBI.41'42  This  comparison  may  be 
particularly  valuable  when  making  decisions  on  whether  a  Service  member  has  recovered  to  his 
or  her  baseline  cognitive  functioning  and  is  fit  to  return  to  duty.  While  this  approach  to 
evaluating  mTBI  has  historically  been  accepted  as  the  standard  of  care,  obtaining  baselines  for 
all  deploying  Service  members  is  expensive,  and  the  overall  impact  on  clinical  decision  making 
is  unclear.  Some  recent  publications  suggest  that  comparison  to  properly  stratified  normative 
data  may  be  sufficient  to  determine  whether  an  individual  can  return  to  duty.7'16  While  fiscally 
advantageous,  the  use  of  normative  values  also  has  drawbacks.  Compared  with  the  use  of 
individual  baseline  values,  using  normative  values  typically  reduces  the  specificity  of  the 
ANAM4  in  detecting  cognitive  deficits  in  individuals.43 

In  the  current  fiscal  environment,  the  interest  in  using  normative  values  versus  individual 
baseline  values  for  comparison  has  increased.  As  the  size  of  the  ANAM  data  repository  has 
expanded,  there  may  be  an  opportunity  to  create  a  more  robust  normative  database  with 
significantly  improved  test  characteristics.  Additionally,  since  more  than  80  percent  of  TBIs  are 
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diagnosed  in  garrison,  having  a  more  accurate  normative  database  may  help  avoid  the  cost 
associated  with  obtaining  baseline  tests  in  all  Service  members.44 


On  July  25,  2014,  the  Undersecretary  for  Personnel  and  Readiness  (USD(P&R))  tasked  the 
Defense  Health  Board  (DHB)  to  review  the  scientific  evidence  for  using  population  normative 
values  for  post-concussive  computerized  neurocognitive  assessments.  From  July  2007  to  July 
2014,  DoD  completed  more  than  1.5  million  pre-deployment  baseline  ANAM  assessments,  of 

Q 

which  an  initial  subset  of  107,000  comprises  the  current  normative  dataset.  As  of  July  2014, 
there  have  also  been  26,524  post-injury  comparison  assessments.5  Given  the  size  of  the  current 
data  repository  and  recent  publications  indicating  that  using  a  normative  data  set  for  comparison 
may  be  as  effective  as  relying  on  individual  baselines  in  assessing  cognitive  function  following 
mTBI,  questions  have  been  raised  about  the  utility  of  continuing  to  collect  pre-deployment 
baselines.  To  address  these  concerns,  the  USD(P&R)  requested  that  the  DHB  examine  the  state 
of  the  science  on  neurocognitive  assessments  and  provide  recommendations  regarding  the  need 
for  continued  baseline  testing,  the  adequacy  of  the  current  normative  dataset,  the  utility  of 
expanding  the  use  of  neurocognitive  testing  beyond  the  deployment  cycle,  areas  for  future 
research,  and  the  cost-benefit  tradeoffs  of  performing  baseline  testing  on  deploying  Service 
members. 


The  Neurological/Behavioral  Health  Subcommittee  of  the  DHB  conducted  an  extensive  literature 
review,  received  briefings,  and  conducted  panel  discussions  with  subject  matter  experts  to 
address  the  questions  outlined  in  the  tasking.  The  Guiding  Principles  (listed  below)  were 
adopted  as  a  foundation  for  review  of  the  evidence  for  baseline  neurocognitive  testing  using  the 
Automated  Neurocognitive  Assessment  Metrics  (ANAM)  4. 


Overarching  Principle 

It  is  the  duty  of  DoD  to  provide  high-quality  care  to  Service  members  who  experience 
mTBI. 

Guiding  Principles 

These  principles  require  that  the  changes  recommended  by  the  subcommittee,  when 
taken  as  a  whole,  must 

i.  ensure  that  Service  members  who  experience  mTBI  receive  the  best  care  possible 
including 

a.  prompt  and  accurate  diagnosis  of  mTBI 

b.  accurate  assessment  of  cognitive  deficits  resulting  from  mTBI 

c.  appropriate  recommendations  for  return-to-duty  post-mTBI; 

ii.  identify  the  most  effective  applications  of  computerized  neurocognitive  testing  in 
assessing  mTBI; 

iii.  be  evidence-based,  taking  into  account  the  most  current  research  and  best 
practices  in  DoD,  other  federal  agencies,  academia,  and  the  private  sector. 
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2.  Automated  Neuropsychological  Assessment  Metrics  4 
History  and  Use 

The  Automated  Neuropsychological  Assessment  Metrics  4  (ANAM4)  is  part  of  a  family  of 
neurocognitive  assessment  tools  known  as  computerized  neuropsychological  assessment  devices 
(CNADs).  The  American  Academy  of  Clinical  Neuropsychology  and  the  National  Academy  of 
Neuropsychology  define  a  CNAD  as  “any  instrument  that  utilizes  a  computer,  digital  tablet, 
handheld  device,  or  other  digital  interface  instead  of  a  human  examiner  to  administer,  score,  or 
interpret  tests  of  brain  function  and  related  factors  relevant  to  questions  of  neurologic  health  and 
illness.”45  The  Department  of  Defense  (DoD)  uses  the  term  “Neurocognitive  Assessment  Tool” 
(NCAT)  to  refer  to  computerized  neurocognitive  assessment  devices.  This  report  will  use  the 
term  NCAT  to  remain  consistent  with  terminology  used  by  DoD.  NCATs  are  widely  used  to 
assess  cognitive  functioning  in  individuals  with  conditions  that  affect  their  brain  function,  such 
as  dementia,  stroke,  lupus,  and  traumatic  brain  injury  (TBI).  Tests  that  are  frequently  used  to 
assess  mild  traumatic  brain  injury  (mTBI)  include  the  Immediate  Post-Concussion  Assessment 
and  Cognitive  Testing  (ImPACT),  ANAM4,  CogSport,  CNS  Vital  Signs,  and  HeadMinders.6'46 
These  tests  rely  on  similar  testing  paradigms  and  demonstrate  no  significant  difference  in 
reliability,  sensitivity,  and  specificity.6’23 

Experts  in  the  field  generally  agree  that,  in  theory,  having  a  valid  baseline  for  computerized 
testing  provides  the  most  accurate  measurement  for  comparison  when  assessing  an  individual  for 

i  o 

mTBI,  particularly  when  the  baseline  and  post-injury  assessments  are  performed  under  similar 
conditions.  A  benefit  of  NCATs  is  that  they  allow  for  accurate,  rapid  analysis  of 
psychometrics  such  as  simple  reaction  time  and  executive  function.13’45  This  is  particularly 
useful  when  patients  have  a  valid  baseline  test  that  accurately  reflects  their  neurocognitive 
function  when  healthy.  While  NCATs  have  improved  the  ability  to  more  efficiently  and 
accurately  assess  time-dependent  cognitive  functions  such  as  simple  reaction  time,  they  are  only 
one  component  of  a  comprehensive  clinical  evaluation  of  mTBI.13’47  Typically,  the  evaluating 
clinician  will  document  the  patient’s  mental  and  physical  symptoms  and  conduct  a  thorough 
survey  of  the  patient’s  mood,  sleep  cycle,  balance,  physical  coordination,  pain,  and  cognitive 
functioning.  An  NCAT  allows  the  clinician  to  determine  if  there  is  evidence  of  an  objective 
change  in  the  patient’s  cognitive  function  by  comparing  their  post-injury  test  results  with  their 
own  pre-injury  baseline  or  with  normative  data  to  determine  if  their  function  is  within  the  normal 
range  of  an  appropriate  comparison  group.  These  data  can  provide  important  insight  to  an 
individual’s  cognitive  function;  however,  the  psychometrics  measured  by  NCATs  are  inherently 
variable  when  other  factors  such  as  fatigue,  stress,  and  mood  are  present.  As  such,  a  clinician 
must  interpret  NCAT  scores  as  part  of  a  broader  examination.  The  frequent  presence  of 
comorbidities  such  as  posttraumatic  stress  disorder  (PTSD),  sleep  disorders,  and  depression  can 
affect  NCAT  test  results  as  well,  requiring  a  nuanced  interpretation  of  test  results  in  the  greater 
context  of  a  patient’s  overall  mental  and  physical  health. 

Given  Terms  of  Reference  (Appendix  B)  for  this  report,  the  discussion  will  focus  on  ANAM4 
and  its  use  in  the  military. 
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2. 1  Role  of  NCATs  in  Mild  Traumatic  Brain  Injury 
Evaluation 

NCATs  are  relatively  new  in  the  field  of  concussion  testing.  The  first  NCATs  were  developed  in 
the  late  1980s  and  were  initially  used  in  the  diagnosis  and  treatment  of  concussion  during  the 
mid  to  late  1990s,  although  their  use  was  not  widespread.  Over  the  past  several  years,  their  use 
has  gained  popularity  in  both  professional  and  amateur  sports  organizations  in  addition  to  their 
use  in  pre-deployment  baseline  testing  of  Service  members.9  The  National  Football  League  has 
instituted  mandatory  baseline  testing  for  its  athletes,  and  many  college,  high  school,  and 
elementary  school  sports  programs  have  begun  to  require  participation  in  NCAT  programs.  In 
spite  of  this  increased  use  for  mTBI  assessment  in  sports,  many  clinicians  do  not  use  NCATs  on 
a  regular  basis.  A  2014  survey  conducted  by  Rabin  et  al  found  that  45.5  percent  of  clinicians 
who  responded  to  the  questionnaire  had  never  used  NCATs,  noting  that  one-on-one  interaction 
with  their  patients  during  the  administration  of  traditional  neurocognitive  testing  is  beneficial  in 
assessing  their  mental  and  cognitive  state.  Additionally,  the  survey  indicated  that  14.1  percent  of 
clinicians  used  them  rarely,  19.6  percent  used  them  sometimes,  18.2  percent  used  them  often, 

48 

and  only  2.6  percent  always  used  them. 

While  the  use  of  NCATs  is  increasing,  questions  remain  on  the  most  appropriate  way  to 
incorporate  their  use  in  assessing  mTBI.  Although  some  organizations  acknowledge  that 
neuropsychological  testing  is  useful  in  evaluating  mTBI,  they  note  that  interpretation  of  NCAT 
test  results  should  be  done  by  a  health  care  professional  with  appropriate  training  and  in  the 
context  of  a  comprehensive  clinical  exam.  Several  key  sports  organizations  have  published 
position  statements  regarding  the  management  of  mTBI  and  the  use  of  computerized  testing  in 
making  return-to-play  decisions.  The  American  Medical  Society  for  Sports  Medicine  notes  the 
following  in  its  2013  position  statement  on  Concussion  in  Sport: 

Most  concussions  can  be  managed  appropriately  without  the  use  of  neuropsychological 
testing...  neuropsychological  tests  are  an  objective  measure  of  brain-behavior 
relationships  and  are  more  sensitive  for  subtle  cognitive  impairment  than  clinical 
exam... Computerized  neuropsychological  testing  should  be  interpreted  by  healthcare 
professionals  trained  and  familiar  with  the  type  of  test  and  the  individual  test  limitations, 
including  a  knowledgeable  assessment  of  the  reliable  change  index,  baseline  variability, 
and  false  positive  and  false  negative  rates. 49(Pg' 2) 

Similarly,  the  2013  consensus  statement  published  after  the  fourth  International  Conference  on 
Concussion  in  Sport  states 

...it  must  be  emphasized,  however,  that  [neuropsychological]  assessment  should  not  be 
the  sole  basis  of  management  decisions.  Rather,  it  should  be  seen  as  an  aid  to  the  clinical 
decision  making  process  in  conjunction  with  a  range  of  assessments  of  different  clinical 
domains  and  investigational  results.  It  is  recommended  that  all  athletes  should  have  a 
clinical  neurological  assessment  (including  assessment  of  their  cognitive  function)  as  part 
of  their  overall  management.  This  will  normally  be  performed  by  the  treating  physician 
often  in  conjunction  with  computerised  neuropsychological  screening  tools.  Formal  NP 
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[neuropsychological]  testing  is  not  required  for  all  athletes;  however, 
when  this  is  considered  necessary,  it  should  ideally  be  performed  by  a  trained 
neuropsychologist. 14 

2.2  History  and  Development  of  the  ANAM 

Development  of  the  ANAM  test  battery  began  in  1984  when  a  team  of  DoD  neuropsychologists 
and  researchers  created  a  method  to  assess  the  impact  of  various  drugs,  medications,  and  nerve 
agents  on  the  cognitive  performance  of  Service  members.10  The  resulting  test  battery  provided  a 
basis  for  understanding  effects  of  medication  and  other  environmental  factors  on  the  brain,  but  it 
was  designed  for  the  assessment  of  long-term  comorbidities  associated  with  mTBI  and  post- 
traumatic  stress.  The  need  for  a  neuropsychological  assessment  tool  was  again  underscored  in 
the  early  1990s  when  Service  members  returned  home  from  Gulf  War  deployments  with  various, 
persistent  neuropsychological  symptoms.  The  Office  of  Military  Performance  Assessment 
Technology  resumed  research  on  the  existing  testing  battery,  developing  the  first  iteration  of 
ANAM  in  1995.  This  first  version  of  ANAM  was  intended  to  detect  changes  in  cognitive 
function  due  to  exposure  to  chemical  agents  and  environmental  factors. 

ANAM  development  paralleled  similar  efforts  in  the  civilian  sports  sector.  Pencil  and  paper 
neuropsychological  testing  had  been  gaining  popularity  as  a  method  to  monitor  and  assess 
neurodegenerative  disease  in  the  1960s,  and  during  the  1970s  researchers  began  studying  its  use 
to  assess  mTBI.51  Clinicians  began  to  use  acute  sideline  and  recovery  assessment 
neuropsychological  testing  for  mTBI  in  the  1970s  when  the  Colorado  Medical  Association 
issued  guidelines  on  concussion  management  in  response  to  the  deaths  of  two  high  school 
football  players  from  head  trauma-related  complications.  It  remained  a  small  field  until  1986, 
when  Barth  et  al  published  a  new  methodology  for  assessing  sports-related  mTBI  that  relied  on 
baseline  testing  to  determine  the  extent  of  the  injury  and  measure  recovery.  This  methodology 
spurred  the  development  of  new  neurocognitive  tests  during  the  1990s  and  drove  the  growth  of 
the  computerized  neuropsychological  testing  industry  to  assess  mTBI  in  sports,  where  baseline 
testing  was  quickly  incorporated  into  preseason  screening  and  concussion  management 
guidelines.  During  this  surge  in  mTBI  testing  technology  in  the  civilian  sector,  DoD  was 
developing  similar  technologies  tailored  to  the  military  population  as  it  created  the  ANAM 
testing  battery  and  began  to  repurpose  it  for  application  as  a  concussion  assessment  tool. 

When  substantial  numbers  of  Service  members  returned  from  Operation  ENDURING 
FREEDOM  (OEF)  and  Operation  IRAQI  FREEDOM  (OIF)  deployments  with  TBI,  there  was 
increased  interest  in  having  an  effective  neurocognitive  assessment  testing  battery.  Diagnosis 
and  treatment  of  these  injuries  was  poorly  understood  and  DoD  required  a  reliable  quantifiable 
method  to  assess  the  impact  of  TBI  on  the  cognitive  functioning  of  Service  members  before  they 
were  allowed  to  return  to  duty.  Rather  than  develop  an  entirely  new  testing  platform,  DoD 
repurposed  the  ANAM  testing  battery,  updating  testing  methods  for  key  areas  affected  by  mTBI 
including  mathematical  reasoning,  simple  response  time,  delayed  memory,  and  spatial 
reasoning.10 

Although  ANAM  was  initially  developed  by  DoD,  the  license  for  the  test  battery  is  currently 
held  by  the  Center  for  the  Study  of  Human  Operator  Performance  (C-SHOP)  at  the  University  of 
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Oklahoma.  C-SHOP  partnered  with  DoD  in  researching  and  developing  the 
computer  platform  and  quality  assurance  measure  for  ANAM,  and  in  2006  DoD  sold  the  license 
to  C-SHOP,  allowing  it  to  continue  its  work  on  the  test  library.50  This  arrangement  requires  that 
DoD  must  pay  to  use  the  test  for  its  pre-deployment  baseline  ANAM  assessment  program. 
DoD’s  current  TBI  assessment  program  requires  DoD  to  contract  with  C-SHOP  and  an  external 
test  administration  service  to  conduct  the  required  baseline  testing  for  deploying  Service 
members.  The  current  licensing  contract  for  ANAM4  pre-deployment  testing  is  $1,000,000  for 
up  to  100,000  ANAM4  tests  and  $140,000  for  each  additional  20,000  tests.5  The  costs 
associated  with  the  current  ANAM  testing  policy  are  further  discussed  in  Section  5. 

The  ANAM  has  evolved  as  researchers  have  learned  more  about  mTBI.  DoD  currently  uses  the 
ANAM4  Traumatic  Brain  Injury  Battery  -  Military  (ANAM4  TBI-MIL,  referred  to  in  the  report 
as  ANAM4),  a  specialized  subset  of  tests  and  questionnaires  from  the  full  ANAM  library.  The 
included  modules  and  the  functions  they  assess  are  shown  in  Table  1  below. 


Table  1.  ANAM4  TBI  Battery  Sub-tests50 


Module  Name 

Function 

Demographics 

User  Profile 

TBI  Questionnaire 

TBI  History 

Sleepiness  Scale 

Fatigue 

Mood  Scale 

Mood  State 

Simple  Reaction  Time 

Basic  Neural  Processing  (speed/efficiency, 
emphasis  on  motor  activity) 

Code  Substitution  -  Learning 

Associative  Learning  (speed/efficiency) 

Procedural  Reaction  Time  (RT) 

Processing  Speed  (choice,  RT/rule  adherence) 

Mathematical  Processing 

Working  Memory 

Matching  to  Sample 

Visual  Spatial  Memory 

Code  Substitution  -  Delayed 

Memory  (delayed) 

Simple  Reaction  Time  (R) 

Basic  neural  processing  (speed/efficiency) 

Adapted  from  C-SHOP,  2007,  ANAM4  TBI:  User  Manual. 


This  selection  of  tests  is  intended  to  measure  those  cognitive  functions  thought  to  be  most 
affected  by  mTBI.  However,  the  psychometrics  used  are  also  inherently  variable  in  humans.50'52 
Factors  such  as  effort,  mood,  sleepiness,  testing  environment,  stress,  and  comorbid  conditions 
can  influence  test  results.  DoD’s  ANAM  program  has  several  measures  in  place  to  control  for 
this  during  pre-deployment  baseline  testing.  For  example,  testing  facilities  are  designed  to 
reduce  distraction,  and  exam  proctors  are  trained  to  review  a  Service  member’s  responses  to  the 
TBI  Questionnaire,  Sleepiness  Scale,  and  Mood  Scale  to  determine  whether  he  or  she  should  be 
able  to  perform  at  his  or  her  normal  level.  ANAM4  proctors  are  also  trained  to  monitor  Service 
members  as  they  take  the  test  to  look  for  signs  of  sleepiness  or  lack  of  effort.50  In  some  cases, 
proctors  have  the  option  to  run  additional  analyses  on  ANAM4  results  to  identify  patterns 
associated  with  intentionally  low  effort  or  the  influence  of  stress,  sleepiness,  or  other  factors. 
Taken  together,  these  measures  support  the  collection  of  accurate  test  results  and  valid  baselines. 

Another  significant  influence  on  test  scores  is  the  practice  effect,  a  “gain  in  score  on  cognitive 
tests  that  occurs  when  a  person  is  retested  on  the  same  instrument,  or  tested  more  than  once  on 
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very  similar  ones.”54  If  a  Service  member  has  taken  the  ANAM4  multiple 
times,  their  score  may  increase  because  of  practice  effects,  possibly  affecting  the  diagnostic 
utility  of  the  test.  A  study  conducted  by  McCrea  et  al  showed  that,  while  the  impact  is  small, 
practice  effects  do  affect  test  scores  in  both  injured  and  uninjured  athletes  when  computerized 
neurocognitive  tests  were  administered  serially  after  mTBI  to  assess  recovery  progress.  Practice 
effects  were  smaller  in  acutely  injured  athletes  than  in  uninjured  athletes,  but  the  effects 
increased  when  tests  were  administered  repeatedly  within  a  short  period  of  time.  NCATs  aim 
to  minimize  the  impact  of  practice  effects  on  serial  test  administrations  through  the  development 
of  multiple  test  forms  to  reduce  predictability  of  the  test.50 

2.3  DoD  AN  AM  Testing  Policy/  Application  op  AN  AM 

The  current  DoD  policy  for  neurocognitive  testing  was  established  in  2008  in  response  to 
congressional  mandate,  outlined  in  Section  1673  of  the  National  Defense  Authorization  Act 
(NDAA)  HR  4966.  The  law  states  “the  Secretary  [of  Defense]  shall  establish. . .  a  protocol  for 
the  pre-deployment  assessment  and  documentation  of  the  cognitive  (including  memory) 
functioning  of  a  member  who  is  deployed  outside  the  United  States  in  order  to  facilitate  the 
assessment  of  the  post-deployment  cognitive  (including  memory)  functioning  of  the  member. 
[The  protocol]  shall  include  appropriate  mechanisms  to  permit  the  differential  diagnosis  of 
traumatic  brain  injury  in  members  returning  from  deployment  in  a  combat  zone.”55  The  law  also 
states  that  the  Secretary  of  Defense  shall  ensure  the  implementation  and  evaluation  of  up  to  three 
pilot  projects  to  obtain  pre-deployment  baselines  to  determine  the  best  method  of  baseline 
collection  and  post-deployment  mTBI  assessment.  ANAM  was  chosen  to  fulfill  this 
requirement,  and  starting  in  2008  all  deploying  Service  members  were  required  to  undergo 
baseline  neurocognitive  testing  within  12  months  prior  to  deployment.  The  baselines  obtained 
from  this  program  are  stored  in  a  data  repository  for  access  if  a  Service  member  experiences 
mTBI  in  the  field  or  reported  past  mTBI  when  completing  the  post-deployment  questionnaire. 

As  the  ANAM  testing  program  matured,  additional  DoD  Instructions  and  Directive  Type 
Memorandums  outlining  standard  procedures,  training,  and  courses  of  action  in  the  identification 
and  treatment  of  TBI  and  mTBI  were  developed  to  further  improve  the  TBI  system  of  care.56'57 
These  guidelines  outline  steps  for  the  medical  management  of  mTBI  and  stress  the  importance  of 
neurocognitive  testing  in  assessing  the  extent  of  the  injury  and  measuring  progress  in  recovery 
post-injury.  The  current  DoD  policy  on  neurocognitive  assessments,  described  in  DoD 
Instruction  Number  6490.13,  states: 

Neurocognitive  assessment  tools  will  be  used  in  a  screening  capacity  to  detect  cognitive 
changes  as  part  of  a  clinical  evaluation  and  will  not  be  used  as  a  standalone  diagnostic 
tool.  The  Automated  Neuropsychological  Assessment  Metrics  (ANAM)  is  the  DoD- 
designated  neurocognitive  assessment  tool  to  support  this  instruction.  The  use  of  the  this 
tool  will  remain  in  effect  until  such  time  as  evolving  science  and  medical  best  practice 

co 

guidelines  inform  a  change  in  policy. 

As  stated  above,  the  requirements  of  the  2008  NDAA  are  in  place,  and  every  Service  member 
who  deploys  undergoes  baseline  testing  within  12  months  prior  to  deployment.  The  baselines 
are  maintained  in  the  ANAM  data  repository  and  clinicians  may  request  access  to  a  Service 
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member’s  baseline  in  case  a  post-injury  or  post-deployment  ANAM 

comparison  is  required.  When  Service  members  redeploy,  ANAM  is  not  administered  unless 
there  is  indication  that  they  may  have  sustained  an  mTBI  while  deployed.  The  ANAM  baseline 
assessment  must  be  re-administered  prior  to  a  subsequent  deployment  if  it  has  been  more  than  12 
months  since  the  previous  baseline  was  established.4 

ANAM4  may  be  used  in  different  ways  to  monitor  recovery  after  an  mTBI.  In  theater,  clinical 
guidelines  suggest  that  the  test  battery  be  administered  if  symptoms  of  concussion  are  present  24 
hours  post-injury.  This  first  administration  should  be  completed  between  24  to  72  hours  post¬ 
injury.1 1  The  test  may  be  re-administered  on  a  regular  basis  as  symptoms  resolve  to  assess  for 
persistent  neurocognitive  deficits  and  inform  the  return-to-duty  decision-making  process.  A 
Service  member’s  in-theater  post-injury  ANAM4  scores  may  be  compared  to  their  individual 
baseline,  if  available,  or  to  the  normative  database  to  determine  whether  cognitive  impairment  is 
present.12 

The  U.S.  Naval  Academy  (USNA)  applies  a  different  methodology  in  its  use  of  ANAM4.  All 
students  at  USNA,  referred  to  as  midshipmen,  undergo  baseline  neurocognitive  testing  with  a 
version  of  ANAM4  utilizing  additional  subtests  for  executive  function,  visual  spatial  processing, 
and  working  memory.  In  addition,  a  built-in  effort  measure  is  used  to  assess  test  validity.  When 
midshipmen  experience  mTBI,  they  are  removed  from  normal  activity  until  their  symptoms 
completely  resolve.  Post-injury  ANAM4  testing  is  administered  when  the  student  is  symptom 
free,  including  after  physical  exertion.  If  a  midshipman’s  ANAM4  score  remains  significantly 
below  baseline  after  physical  symptoms  have  resolved,  he  or  she  will  be  referred  for  additional 

o 

testing  to  guide  further  treatment  and  recovery. 

The  two  approaches  listed  above  for  using  the  ANAM4  to  inform  return-to-duty  decisions 
embody  significant  differences  in  the  perceived  value  of  baseline  testing.  The  first  approach, 
applied  in  theater,  aims  at  documenting  the  occurrence  of  mTBI  and  measuring  acute  cognitive 
impairment  as  part  of  the  larger  clinical  assessment  to  inform  immediate  management.  Several 
studies  support  this  application  of  the  test  battery.59'60  In  a  study  conducted  by  McCrea  et  al, 
patients  who  experienced  mTBI  underwent  computerized  neurocognitive  testing  on  a  regular 
basis,  starting  in  the  acute  phase  after  their  injuries.  The  study  data  indicate  that  dramatic 
improvement  in  test  scores  occurs  by  72  hours  post-injury  and  that  by  day  7  post-injury  most 
individuals  have  recovered  to  baseline.25  This  is  further  supported  by  Kelly  et  al,  who  note  that 
most  symptoms  and  deficits  tend  to  resolve  by  day  10  post-injury,  and  if  not,  more  in-depth 
analysis  (beyond  computerized  neurocognitive  testing)  is  required.60  6  The  second  approach  is  to 
administer  the  ANAM4  after  a  patient  is  symptom-free  and  is  characterized  by  USNA’s  use  of 
ANAM4,  which  is  intended  to  identify  signs  of  mTBI  that  are  slow  to  resolve  or  that  may  not  be 
obvious  through  other  means  of  post-injury  assessment.  The  ongoing  comparison  to  a 
midshipman’s  individual  baseline,  after  symptom  resolution,  reduces  the  risk  of  false-positive  or 
false-negative  diagnoses  and  prevents  the  midshipman  from  returning  to  full  activities 
prematurely.  This  approach  is  supported  by  a  study  conducted  by  Fazio  et  al  that  used  ImPACT 
to  assess  concussed  and  non-concussed  athletes  one  week  after  sustaining  an  mTBI.  All  of  the 
injured  athletes  demonstrated  significantly  lower  test  scores  than  the  uninjured  controls.  Further, 
injured  athletes  who  reported  full  resolution  of  symptoms  after  one  week  still  scored 
significantly  lower  than  their  uninjured  counterparts,  indicating  that  full  resolution  of  physical 
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symptoms  may  not  indicate  complete  recovery.61  A  study  conducted  by  Haran 

et  al  in  2013  also  supports  the  use  of  ANAM4  beginning  after  the  acute  injury  phase  following 

mTBI  has  passed.  The  study  investigated  the  return  to  neurocognitive  baseline  in  Marines  using 

ANAM4  tests  administered  between  2  and  8  weeks  post-deployment  and  between  3  to  12  months 

post-deployment,  and  found  that  cognitive  deficits  can  persist  for  1  to  3  months  after  the  initial 
•  •  6,62 
injury. 
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3.  Sensitivity,  Specificity,  and  Reliability  of  Computerized 
Neurocognitive  Testing 

Current  DoD  policy  mandates  that  each  Service  member  complete  Automated  Neurocognitive 
Assessment  Metrics  4  (ANAM4)  baseline  testing  within  12  months  prior  to  deployment. 
However,  recent  budget  constraints  in  combination  with  new  research  suggesting  that  normative 
data  may  be  sufficient  to  identify  decreases  in  neurocognitive  function  post-mild  traumatic  brain 
injury  (mTBI)  has  prompted  a  review  of  the  policy  and  its  associated  costs. 

Comparing  the  relative  effectiveness  of  individual  baseline  ANAM4  results  and  normative  data 
in  post-injury  assessments  is  complex.  Most  experts  agree  that,  logically,  a  valid  high-quality 
individual  baseline  would  provide  the  most  accurate  data  for  clinicians  to  make  a  well-informed 
return-to-duty  decision  for  a  given  individual.  However,  recent  research  indicates  that  collecting 
an  accurate  baseline  is  challenging  and  that  many  of  the  available  baselines  in  the  military’s 
ANAM4  data  repository  may  not  be  of  sufficient  quality. 111516  Other  elements  that  must  be 
considered  in  this  comparison  are  the  value  that  baseline  testing  provides  for  individuals  who 
ordinarily  perform  significantly  above  or  below  the  norm;  challenges  in  appropriately 
interpreting  ANAM4  test  results;  and  the  size  and  stratification  of  the  existing  military  ANAM4 
normative  database. 

Several  quantitative  and  qualitative  measures  are  used  in  evaluating  the  accuracy  and 
effectiveness  of  using  individual  baselines  versus  normative  data  for  post-injury  comparison. 
These  measures  include  sensitivity,  specificity,  and  test/re-test  reliability.  It  is  desirable  that  a 
neurocognitive  test  demonstrate  high  sensitivity  and  specificity.  As  defined  by  McCrea  et  al, 

[sensitivity] . .  .refers  to  the  probability  that  an  injured  participant  will  be  identified  as 
“abnormal”  by  a  change  in  test  perfonnance.  At  time  points  subsequent  to  time  of  injury, 
sensitivity  values  indicate  the  probability  that  a  player  originally  injured  continued  to  be 
classified  as  “abnormal”  according  to  at  least  one  of  the  test  measures.  Specificity. . . 
refers  to  the  probability  that  a  control  participant  will  be  correctly  classified  as  “normal” 
using  the  same  method.” 

Test/re-test  reliability  is  an  indicator  of  an  NCAT’s  consistency.  Cole  et  al  define  test/re-test 
reliability  as  “the  association  between  the  scores  obtained  by  a  single  examinee  taking  the  same 
test  on  two  or  more  different  occasions.”6  Reliability  is  usually  expressed  using  Pearson’s  r 
coefficient  or  intraclass  correlation  values.  The  higher  the  r  value  or  intraclass  correlation,  the 
higher  the  NCAT’s  reliability.  Reliable  change  indices  may  also  be  used  to  describe  the 
reliability  of  an  NCAT.  High  test/re-test  reliability  indicates  a  robust  testing  platform  that 
minimizes  or  accounts  for  the  impacts  of  practice  effect,  testing  environment,  psychological 
state,  and  attitude  on  individual  test  scores.6 

3.1  Test/Re-Test  Reliability 

A  key  argument  against  the  continued  use  of  computerized  baseline  testing  is  that  the  reliability 
of  available  test  batteries  is  not  high  enough  to  ensure  that  baselines  are  valid.  ’  Environmental 
factors,  natural  variation  in  cognitive  function,  and  inconsistent  data  interpretation  methods  may 
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preclude  accurate  baseline  comparisons.  “  When  compared  with  other  NCATs  such  as  ImPACT, 
CogState  Sport,  and  CNS  Vital  Signs,  ANAM4  demonstrates  similar  reliability  and  validity.23 
Cole  et  al  conducted  a  study  in  healthy,  active  duty  Service  members,  in  which  study  participants 
were  assigned  to  take  one  of  the  four  NCATs  listed  above  twice  within  a  30-day  period.  They 
found  that,  within  their  healthy  study  population,  most  NCATs  showed  low  intraclass  correlation 
in  both  individual  subtests  and  as  a  whole.  Overall,  they  concluded  that  test-retest  reliabilities 
were  consistent  with  those  previously  reported  in  the  literature  and  “less  than  suggested  for 
clinical  use.”6  One  methodologic  concern  with  the  study  is  that  non- standardized  computers 
were  used  for  testing,  potentially  leading  to  significant  error  in  reaction  time-dependent  subtests.6 

Another  factor  that  affects  test  results  is  mindset.  Some  research  suggests  that  when  individuals 
take  neurocognitive  test  batteries  while  anxious  they  demonstrate  lower  test  scores.  A  2003 
study  by  Suhr  et  al63  explored  a  theory  called  “diagnosis  threat,”  which  is  thought  to  lower 
neurocognitive  test  scores  due  to  anxiety,  depression,  and  increased  effort  associated  with  fear  of 
a  poor  diagnosis.  The  study  observed  test  performance  in  individuals  with  a  history  of  mTBI. 

The  experimental  group  was  informed  that  they  were  being  tested  because  of  their  head  injury. 
They  were  matched  with  controls  who  had  a  similar  mTBI  history,  but  were  not  informed  that 
the  testing  was  related  to  their  injuries.  Suhr  et  al  found  that  the  experimental  group  performed 
worse  than  the  control  group  on  tests  of  delayed  recall,  psychomotor  speed,  working  memory, 
and  attention.  After  assessing  both  groups  for  depression  and  for  effort,  researchers  found  that 
depression  did  not  account  for  the  difference  between  groups  and  that  both  the  control  group  and 
the  experimental  group  exhibited  the  same  level  of  effort.  They  concluded  that  even  the  threat  of 
being  perceived  to  have  cognitive  deficits  because  of  mTBI  could  worsen  performance  on 
neurocognitive  assessments.63 

3.2  Impact  of  Self-reported  mTBI  on  Post-Deployment  Neurocognitive 
Test  Scores 

Some  studies  have  investigated  the  effect  of  self-report  of  TBI  during  deployment  on  post¬ 
deployment  neurocognitive  test  scores.  A  2014  study  by  Cooper  et  al  examined  the  factors 
associated  with  neurocognitive  performance  in  Service  members  who  self-reported  having 
experienced  mTBI  while  deployed  overseas  during  Operation  IRAQI  FREEDOM  (OIF)  or 
Operation  ENDURING  FREEDOM  (OEF).  Service  members  who  enrolled  in  the  study  had 
redeployed  and  reported  persistent  cognitive  difficulties  and  mTBI-related  symptoms.  These 
individuals  took  the  Repeatable  Battery  for  the  Assessment  of  Neuropsychological  Status 
(RBANS)  test  to  measure  their  cognitive  function.  The  researchers  found  that  overall  the 
Service  members  scored  in  the  lower  end  of  the  average  range.  In  analyzing  which  factors  had 
the  greatest  impact  on  the  test  score,  they  found  that  test  taker  effort  had  the  greatest  effect  on 
score,  followed  by  symptom  severity.  They  found  that  a  significant  number  of  study  participants 
failed  the  RBANS  effort  index,  highlighting  the  importance  of  including  a  robust  effort  measure 
when  administering  neurocognitive  assessments.  When  analyzing  the  scores  of  all  Service 
members,  including  those  who  failed  the  effort  index,  self-reported  symptoms  accounted  for  a 
statistically  significant  effect  on  test  scores.  However,  when  analyzing  only  the  test  scores  of 
Service  members  who  demonstrated  sufficient  effort  on  the  RBANS,  it  was  determined  that 
factors  such  as  demographics,  nature  of  the  injury,  and  reported  symptom  severity  had  a 
relatively  small  though  statistically  significant  impact  on  test  scores.64  It  is  important  to  note  that 
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the  study  only  included  individuals  who  were  not  in  the  acute  phase  of  mTBI,  which  may  explain 
why  reported  symptoms  were  not  associated  with  a  statistically  significant  decrease  in  test 
scores. 

A  2009  study  by  Ivins  et  al  further  supports  this  theory.  The  study  examined  a  sample  of  U.S. 
Army  soldiers  who  had  served  in  Iraq  and/or  Afghanistan  to  determine  whether  a  self-reported 
history  of  deployment-related  TBI,  lifetime  history  of  TBI,  and  current  post-concussive  symptom 
status  affected  cognitive  test  performance.65  The  results  did  not  demonstrate  either  an 
“association  between  having  a  history  of  mTBI  and  poor  ANAM  performance”  or  an 
“association  between  poor  ANAM  performance  and  the  reported  number  of  TBIs,  injury 
severity,  and  the  number  of  problematic  post-concussive  symptoms.”  Bruce  and  Echmendia  also 
found  that  there  was  no  correlation  between  poor  scores  and  self-reported  total  number  of 
lifetime  TBIs,  injury  severity,  or  ongoing  symptoms.66 

3.3  Sensitivity  and  Specificity 

The  symptoms  of  mTBI  are  variable  and  poorly  defined.  Individuals  who  have  experienced 
mTBI  manifest  symptoms  differently  depending  on  the  part  of  the  brain  that  was  injured,  making 
the  impact  of  mTBI  on  various  neurocognitive  functions  difficult  to  assess.  Normal  fluctuations 
in  neurocognitive  function  that  occur  because  of  shifts  in  environment,  mood,  and  practice  can 
distort  or  obscure  changes  in  cognitive  function  that  occur  with  mTBI.  These  challenges  can 
decrease  the  sensitivity  and  specificity  of  ANAM4.  Although  ANAM4  is  used  in  conjunction 
with  other  clinical  information  when  assessing  mTBI,  researchers  have  investigated  its 
effectiveness  independently  to  study  how  sensitivity  and  specificity  change  when  post-injury  test 
results  are  compared  with  baselines  versus  normative  data.  A  2013  study  conducted  by  Mihalik 
et  al  compared  serial  tests  of  healthy  athletes  and  concussed  athletes  to  measure  the  sensitivity 
and  specificity  of  ANAM4’s  test  battery.  Using  the  baselines  of  the  healthy  athletes  as  a  norm, 
they  found  that  ANAM4  did  successfully  identify  cognitive  deficits  in  a  small  number  of  athletes 

TO 

who  experienced  mTBI.  However,  they  found  that  sensitivity  was  low  for  both  the  individual 
subtests  and  the  test  battery  as  a  whole.  Sensitivity  increased  to  50  percent  when  a  patient’s 
symptom  score  was  included  in  analysis  of  the  test  results,  but  not  significantly  enough  to  allow 
ANAM4  to  serve  as  an  independent  mTBI  assessment  method.  Mihalik  et  al  found  that  for 
individual  subtests,  as  well  as  the  combined  battery,  specificity  was  high  (greater  than  .80).  This 
indicates  that  while  ANAM4  did  not  demonstrate  high  sensitivity,  it  accurately  identified  deficits 

•5  0 

in  cognitive  function  in  concussed  athletes.  These  findings  are  supported  by  the  research  of 
Broglio  et  al,  who  studied  the  sensitivity  of  NCATs  in  detecting  cognitive  deficits  after  mTBI. 
The  study  found  that  most  NCATs  identify  declines  in  cognitive  function  in  at  least  one  area  for 
most  concussed  individuals.15  However,  this  level  of  sensitivity  was  only  achieved  when  test 
results  were  interpreted  in  conjunction  with  symptom  severity. 

A  2012  study  by  Bryan  et  al  that  investigated  the  neurocognitive  performance  of  Service 
members  who  experienced  mTBI  while  deployed  found  that  mTBI  does  affect  neurocognitive 
performance.67  Unlike  the  previous  studies,  this  study  examined  the  ANAM4  tests  scores  from 
neurocognitive  assessments  completed  in  theater  as  part  of  the  initial  injury  assessment,  which 
included  a  full  physical  and  psychological  evaluation.  The  results  indicated  that  service 
members  with  mTBI  showed  greater  declines  in  some  speed  and  throughput  subtests  compared 
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to  service  members  without  TBI.  No  significant  differences  in  accuracy  were  noted.  The 
authors  concluded  that  the  findings  “support  the  use  of  the  ANAM  as  a  clinical  screening  tool 
among  at-risk  populations,  such  as  those  with  a  positive  history  of  TBI,”  and  that  focusing  on 
reaction  times  “may  serve  as  a  more  sensitive  measure  of  cognitive  decline.” 

A  2001  study  conducted  by  Echemendia  et  al  investigated  the  ability  of  several 
neuropsychological  test  batteries  to  detect  declines  in  cognitive  function  following  mTBI.  The 
study  model,  conducted  in  college-level  athletes,  collected  baseline  data  for  all  athletes.  Each 
athlete  was  paired  with  a  control,  and  both  received  a  follow-up  neurocognitive  assessment 
following  mTBI  that  occurred  during  game  play.  Students  who  sustained  mTBI  were  tested  2- 
hours,  48-hours,  1-week,  and  1-month  post-injury  for  cognitive  impairment.  Data  indicated  that 
injured  athletes  showed  cognitive  impairment  for  the  2-hour,  48-hour,  and  1-week  assessments. 
At  the  1 -month  mark,  the  control  group  and  the  injured  group  were  indistinguishable,  and  both 
were  performing  at  levels  higher  than  their  baseline,  suggesting  that  there  may  be  a  practice 
effect.  The  authors  concluded  that  baselines  may  be  useful;  however,  neurocognitive 
assessments  are  highly  variable  and  should  only  be  considered  as  part  of  a  comprehensive 
evaluation.41  This  study  also  indicates  that  computerized  neurocognitive  tests  demonstrate 
sufficient  sensitivity  and  specificity  in  detecting  neurocognitive  impairments  after  mTBI  when 
acute  phase  post-injury  tests  are  compared  to  a  baseline. 

Relatively  few  studies  examine  how  sensitivity  and  specificity  shift  depending  on  whether 
individual  baselines  or  normative  data  are  used  for  comparison  in  a  post-mTBI  assessment.  A 
study  conducted  by  Schmidt  et  al  in  2010  compared  the  sensitivity  and  specificity  of  ANAM4 
using  gender  norms  and  baseline  values  for  comparison.  One  thousand  sixty  college  athletes 
underwent  ANAM4  testing  that  included  initial  and  end-of-test  simple  reaction  time  assessments 
to  assess  fatigue  after  mental  exertion.  Six  hundred  thirty-eight  healthy  athletes  served  as 
matched  controls  for  the  258  individuals  who  experienced  mTBI,  and  175  of  the  injured 
individuals  had  pre-injury  baselines  for  comparison.  After  analyzing  each  subtest  and  the  battery 
as  a  whole  using  both  baselines  and  gender-specific  normative  data,  they  found  that  the  baseline 
comparison  method  identified  2.6  times  as  many  impairments  as  the  normative  method  in  the 
initial  simple  reaction  time  test.  Normative  data  identified  7.6  times  as  many  impairments  in 
mathematical  processing.  For  all  other  measures,  the  baseline  comparison  and  normative  data 
were  in  statistical  agreement.  The  authors  state  that  while  these  differences  are  significant,  the 
overall  sensitivity  of  each  subtest  to  mTBI  must  be  taken  into  account.  Certain  subtests,  notably 
simple  reaction  time,  appear  to  demonstrate  higher  sensitivity  to  mTBI  than  others. 11,68  The 
authors  note  that  the  overall  number  of  impairments  for  the  first  simple  reaction  time  test  was  so 
low  that,  when  the  test  is  administered  on  a  large  scale,  the  relative  improvement  seen  in 
detecting  declines  in  the  first  simple  reaction  time  test  using  baseline  as  opposed  to  normative 
comparisons  was  minor.7 

Echemendia  conducted  another  study  comparing  the  effectiveness  of  the  normative  method 
versus  the  baseline  method  for  assessing  mTBI  using  ImPACT.  Using  a  population  of  223 
college  athletes  with  ImPACT  baselines,  they  compared  two  approaches  for  calculating  reliable 
change  for  the  baseline  method,  finding  that  the  Gulliksen-Lord-Novick  (GLN)  method 
consistently  identified  more  injured  subjects  than  the  Jacobson  and  Truax  approach.  They  then 
compared  the  number  of  athletes  identified  as  cognitively  impaired  as  a  result  of  their  injuries 
using  the  GLN  method  versus  the  normative  method.  They  found  significant  agreement  between 
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the  methods,  noting  that  the  majority  of  injured  athletes  were  correctly  identified  using 
normative  data  alone.  The  authors  concluded  that  “the  majority  of  collegiate  athletes  who 
experience  clinically  meaningful  post-concussion  cognitive  decline  can  be  identified  without 
baseline  data.”69 

In  contrast  to  the  findings  of  Schmidt  et  al,  Echemendia,  and  Register-Mihalik,  a  2014  study  by 
Louey  et  al  found  that  the  baseline  method  for  assessing  mTBI  is  more  sensitive  than  the 
normative  data  method.  Using  norms  stratified  by  age  and  education,  they  found  that  both 
methods  were  highly  specific  and,  overall,  both  methods  correctly  classified  subjects  as  healthy 
or  injured.  However,  when  the  injured  athletes  identified  by  the  baseline  method  were  compared 
to  the  number  identified  by  the  normative  method,  they  found  that  27.6  percent  of  the  injured 
athletes  who  had  been  identified  using  the  baselines  method  were  not  identified  using  normative 
data.43 

The  findings  of  Louey  et  al  are  supported  by  a  2015  study  conducted  by  Hinton-Bayre.70  In  this 
study,  Hinton-Bayre  reviewed  the  studies  published  by  Echemendia,  Schmidt,  and  Louey,  noting 
that  the  study  designs  and  analyses  employed  in  the  Schmidt  and  Echemendia  studies  were 
flawed  and  resulted  in  inaccurately  high  estimates  of  the  sensitivity  and  specificity  of  normative 
data.  Hinton-Bayre  also  expressed  concern  regarding  the  use  of  computerized  neurocognitive 
testing,  preferring  the  use  of  traditional  pencil  and  paper  assessments  to  compare  the  baseline 
versus  normative  data  approaches  to  evaluating  mTBI.  In  a  study  of  professional  rugby  players, 
Hinton-Bayre  found  that  if  individual  baselines  are  not  available,  normative  data,  stratified  by 
age,  ethnicity,  IQ  and  previous  mTBIs,  does  contribute  beneficially  to  the  management  of  mTBI. 
However,  individual  baselines  were  much  more  informative,  accounting  for  intra-individual 
variation  and  reducing  the  occurrence  of  false  positive  and  false  negative  classifications.  As 
with  the  Louey  study,  it  should  be  noted  that  the  sample  size  for  this  study  was  small,  comparing 
only  27  injured  players  with  26  uninjured  controls. 

There  are  many  challenges  associated  with  assessing  the  impact  of  test-retest  variability  in 
accurately  interpreting  results  obtained  from  computerized  NCATs.  ’  ’  This  inherent 
variability  complicates  analysis  of  post-mTBI  test  results,  especially  in  cases  when  a  Service 
member’s  healthy  baseline  ANAM4  test  score  is  significantly  above  or  below  the  average.71  The 
Service  members  who  fall  within  the  tails  of  the  distribution  face  a  greater  risk  of  misdiagnosis 
and  incorrect  return-to-duty  decisions,  because  normative  data  would  not  provide  a  valid 
comparison  to  determine  when  these  individuals  returned  to  the  normal  range  of  function. 

Multiple  subject  matter  experts  indicated  that  these  cases  (in  the  tails  of  the  distribution)  pose  a 
significant  challenge,  especially  when  an  individual’s  baseline  is  not  available.  Service  members 
who  test  significantly  above  or  below  average  when  healthy  run  a  significantly  higher  risk  of 
misdiagnosis  when  normative  data  are  used  in  analysis  of  ANAM4  results.  If  Service  members 
who  test  above  average  experience  an  mTBI  with  the  associated  symptomatology  and  decline  in 
cognitive  function,  comparison  of  their  post-injury  ANAM4  scores  with  normative  data  may  not 
indicate  that  their  cognition  is  compromised,  as  their  ANAM4  score  may  have  decreased  from 
above  average  to  the  norm.  The  resulting  false-negative  diagnosis  may  result  in  the  Service 
member  being  returned  to  duty  prematurely,  potentially  placing  them  and  their  mission  at  risk, 
while  also  leaving  a  persistent  decrement  in  cognitive  function  unrecognized. 
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Conversely,  Service  members  who  would  test  below  average  on  the  ANAM4  when  healthy  run 
the  risk  of  false-positive  diagnoses  when  their  post-mTBI  ANAM4  scores  are  compared  to 
normative  data.  Without  a  previous  healthy  baseline,  these  individuals  will  demonstrate  below 
average  cognitive  function  consistent  with  mTBI;  however,  as  their  symptoms  resolve  over  time, 
even  if  their  ANAM4  scores  improve  to  what  would  be  their  normal  baseline  function,  it  will 
appear  as  if  they  have  a  persistent  deficit  in  cognitive  function  relative  to  the  norm.  A  2013 
study  by  Roebuck-Spencer  et  al  highlights  this  phenomenon  and  endorses  the  value  of  baseline 
testing.  It  compared  the  rates  of  atypical  (potentially  impaired)  performance  on  ANAM4  when 
soldiers’  post-deployment  test  scores  were  compared  with  normative  data  and  their  individual 
baselines.  They  found  that  both  methods  identified  atypical  results  at  a  similar  rate.  However, 

“a  high  percentage  of  individuals  classified  as  atypical”  using  normative  data  “showed  no  change 
from  baseline,  indicating  that  using  norms  alone  may  result  in  a  large  number  of  false  positive 
errors.”  The  resulting  misclassification  can  result  in  increased  costs  and  use  of  medical 
resources. 

Finding  1.1  :  Current  evidence  is  inconclusive  on  whether  using  individual  baseline 
computerized  neurocognitive  assessment  test  results  is  more  advantageous,  on  a 
population  level,  than  using  an  optimally  stratified  normative  dataset  in  evaluating  and 
managing  mild  traumatic  brain  injury,  including  assessing  return  to  duty  decision  making 
and  prognosis. 

Finding  1.2  :  The  current  ANAM4  military  normative  dataset  is  stratified  solely  by  age 
and  sex  and  does  not  accurately  estimate  baseline  neurocognitive  function  in  individuals 
who  score  at  the  high  or  low  ends  of  the  scale.  Efforts  are  currently  underway  to  expand 
and  further  stratify  this  dataset  to  improve  accuracy  in  estimating  cognitive  deficits 
related  to  mild  traumatic  brain  injury. 

Recommendation  1:  DoD  should  continue  to  analyze  existing  ANAM4  data  to 
determine  whether  an  optimally  stratified  normative  dataset  can  be  developed  that  is 
capable  of  accurately  estimating  baseline  neurocognitive  function,  including  for 
individuals  who  score  at  the  high  or  low  ends  of  the  scale.  Specific  stratification 
variables  to  consider  adding  include  education,  rank,  standardized  test  scores, 
race/ethnicity,  and  environmental  factors  such  as  socioeconomic  status. 

Finding  2:  It  is  not  apparent  that  an  adequate  overall  assessment  of  the  utility  of  the 
current  ANAM4  testing  program  in  evaluating  and  managing  mild  traumatic  brain  injury 
has  been  accomplished.  Moreover,  it  is  not  apparent  that  clinical  evaluation  and 
disposition  data  have  been  centrally  compiled  to  accomplish  this  analysis. 

Recommendation  2:  DoD  should  analyze  current  clinical  data  from  mild  traumatic 
brain  injury  evaluations  to  determine  in  what  proportion  of  cases  ANAM4  testing 
provided  information  of  clinical  value  that  was  a  contributing  factor  in  overall 
management  and  disposition  of  the  patient.  As  part  of  this  analysis,  the  value  of  having 
an  individual  baseline  for  comparison  should  be  assessed. 
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Finding  3  :  Current  evidence  is  inconclusive  regarding  the  value  of  routine  pre¬ 
deployment  baseline  ANAM4  testing. 

Recommendation  3:  DoD  should  request  that  legislation  requiring  routine  pre¬ 
deployment  neurological  cognitive  assessments  be  rescinded  to  allow  discontinuation  of 
routine  pre-deployment  ANAM4  baseline  testing.  DoD  should  instead  conduct 
prospective  research  using  neurocognitive  or  other  assessment  tools  to  evaluate  their 
usefulness  in  the  management  of  mild  traumatic  brain  injury  and  return-to-duty  decision¬ 
making. 

3.4  Effect  of  ANAM4/NCATs  on  Clinical  Decision  Making 

There  is  little  research  on  the  impact  of  ANAM4  on  clinical  decision  making  and  patient 
outcomes.  A  study  conducted  by  Kelly  et  al  in  2012  investigated  the  impact  of  ANAM4  use  on 
the  ability  of  clinicians  to  correctly  identify  cognitive  deficits.  The  researchers  compared  the 
sensitivity  and  specificity  with  which  clinicians  were  able  to  identify  decreases  in  procedural 
reaction  time  (PRT)  and  simple  reaction  time  (SRT)  performance  when  using  ANAM  or 
traditional  methods  when  assessing  patients  who  experienced  loss  of  consciousness  or  post- 
traumatic  amnesia  due  to  mTBI.  They  found  that  using  ANAM4  increased  the  discriminant 
ability  of  clinicians  to  detect  changes  in  SRT  and  PRT.  Using  ANAM4  did  not  increase 
specificity,  but  it  did  increase  the  sensitivity,  allowing  clinicians  to  more  easily  identify  patients 
experiencing  cognitive  deficits.11  Further  research  is  needed  into  how  clinicians  use  ANAM4 
and  how  much  it  affects  their  decisions  in  managing  mTBI;  however,  this  study  demonstrates 
that  computerized  neurocognitive  testing  does  provide  beneficial  information  in  the  overall 
assessment  of  mTBI. 

A  study  conducted  by  Van  Kampen  et  al  in  2006  investigated  the  role  of  neurocognitive  testing 
in  athletes  following  sports-related  mTBI.  The  researchers  used  ImPACT  to  collect  pre-season 
baselines  and  post-injury  and  post-season  test  scores  for  high  school  and  college  athletes.  Study 
participants  were  asked  to  report  their  symptoms  in  addition  to  completing  ImPACT  testing.  The 
study  found  that  athletes  were  less  likely  to  be  correctly  identified  as  having  an  mTBI  when 
diagnoses  relied  solely  on  reported  symptoms,  but  when  post-injury  ImPACT  scores  were 
analyzed,  a  greater  number  of  injured  athletes  were  correctly  identified.  The  authors  note  that 
this  is  indicative  of  “an  imperfect  agreement  between  self-reported  symptoms  and  decreased 
neurocognitive  test  scores  after  concussion.”  Van  Kampen’s  work  is  supported  by  Fazio  et  al, 
who  also  found  that  reliance  on  self-reported  symptoms  alone  is  not  as  accurate  as  administering 
neuropsychological  testing  in  addition  to  a  symptom  report.61 

Assessing  prognosis  and  predicting  retum-to-duty/play  time  after  an  mTBI  can  be  difficult. 
Research  indicates  that  there  are  several  predictors  for  prolonged  recovery  including  symptoms 
of  headache  lasting  longer  than  3  hours,  difficulty  concentrating,  retrograde  amnesia,  or  loss  of 

1  Q 

consciousness.  No  studies  on  the  usefulness  of  ANAM4  in  predicting  long-term  prognosis 
were  located,  however,  some  studies  examined  ANAM4  scores  and  retum-to-duty  time.  A  study 
by  Norris  et  al  in  2013  found  a  correlation  between  ANAM4  simple  reaction  time  scores  and 
length  of  recovery  following  an  mTBI.  In  their  retrospective  analysis  of  medical  records  of 
Service  members  who  experienced  mTBI,  they  found  that  the  individuals  who  scored  in  the 
lowest  25  percent  on  the  ANAM4  reaction  time -based  subtests  had  a  median  return-to-duty  time 
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of  19  days,  whereas  those  in  the  upper  25  percent  had  a  median  return-to-duty  time  of 

approximately  7  days.17 

Finding  4:  ANAM4  testing  has  not  yet  been  shown  to  consistently  or  significantly 
contribute  to  assessing  long-term  prognosis  after  a  mild  traumatic  brain  injury. 

Recommendation  4:  DoD  should  analyze  post-injury  test  scores  and  long-term  data 
documenting  the  recovery  of  Service  members  after  a  mild  traumatic  brain  injury  to 
determine  if  post-injury  ANAM4  scores  consistently  or  significantly  contribute  to  the 
assessment  of  prognosis. 
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4.  Automated  Neuropsychological  Assessment  Metrics  4  Normative 
Database  and  Data  Repository 

Since  the  Neurocognitive  Assessment  Tool  (NCAT)  program  was  initiated  in  2008,  the  U.S. 
military  has  collected  more  than  1.8  million  baseline  Automated  Neuropsychological 
Assessment  Metrics  4  (ANAM4)  test  results,  which  are  stored  in  a  data  repository.  An  initial 
normative  database  specific  to  the  U.S.  military  population  was  created  in  2008  with  107,000  of 
these  test  results  for  use  in  evaluating  mild  traumatic  brain  injury  (mTBI).  This  database  is 
currently  stratified  by  age  and  sex  only.5  Experts  in  the  field  of  neuropsychological  testing 
suggest  that  further  stratifying  the  military  normative  database  may  improve  statistical  accuracy 
in  identifying  changes  in  cognitive  function  when  evaluating  mTBI  with  post-injury  ANAM4 
test  scores.  Research  showing  that  demographic  factors  such  as  education,  handedness, 
ethnicity,  and  rank  may  affect  performance  on  neurocognitive  testing19'20  supports  the  need  for  a 
more  highly  stratified  database.  Other  variables  suggested  by  subject  matter  experts  that  may  be 
of  value  include  occupation  and  standardized  testing  scores  (such  as  Armed  Services  Vocational 
Aptitude  Battery).  Addition  of  these  strata  may  require  the  incorporation  of  additional  baselines 
for  underrepresented  subgroups  within  the  current  military  population. 

4. 1  The  Current  Data  Repository 

The  ANAM  program  is  currently  managed  by  the  U.S.  Army  Medical  Department  (AMEDD). 
Until  recently,  ANAM  test  results  were  stored  in  a  data  repository  that  was  not  structured  in  a 
format  conducive  to  statistical  analysis.  Clinicians  could  access  a  Service  member’s  previous 
test  results  by  submitting  a  request  to  a  24/7  ANAM  call  center  via  telephone,  email,  or  fax,  but 
it  was  not  possible  to  easily  access  large  numbers  of  scores  to  conduct  larger-scale  research.  As 
such,  it  was  a  significant  untapped  resource  for  research  on  mTBI,  retrospective  analyses,  and 
development  of  an  improved  normative  database.  In  efforts  to  utilize  this  vast  quantity  of  data, 
the  ANAM  program  office  has  undertaken  the  conversion  of  the  data  repository  into  a  true 
database.  As  of  August  2015,  the  DoD  ANAM  program  office  had  successfully  converted  more 
than  1.8  million  ANAM4  records  into  a  database  format,  identified  1.1  million  records  suitable 
for  use  in  constructing  an  improved  normative  dataset,  and  had  begun  analysis  to  identify 
sources  of  additional  demographic  factors  that  may  be  applied  to  further  improve  the 
stratification  and  accuracy  of  the  normative  dataset.5 

4.2  Maintaining  the  Normative  Database 

The  current  U.S.  military  ANAM4  normative  database  was  created  in  2008.  Since  a  normative 
database  should  reflect  the  population  being  tested73  and  the  demographic  characteristics  of  the 
military  population  will  likely  change  over  time,  it  should  be  updated  on  a  regular  basis.  Thus, 
there  must  be  an  ongoing  effort  to  continually  collect  and  update  the  baselines  to  reflect  the 
current  demographic  distribution  of  the  military  population  to  maintain  a  current  normative 
database.  Since  the  current  ANAM  program  targets  military  members  who  are  scheduled  to 
deploy  for  baseline  testing,  it  is  possible  that  the  database  over  represents  young,  healthy, 
enlisted  males. 
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5.  Automated  Neuropsychological  Assessment  Metrics  Program 
Costs 

5.1  Current  Program  Costs 

In  2008,  Congress  included  a  provision  in  the  National  Defense  Authorization  Act  requiring  all 
deploying  Service  members  to  undergo  pre-deployment  baseline  neurocognitive  testing.  As  an 
initial  response  to  this  requirement,  DoD  elected  to  use  the  Automated  Neuropsychological 
Assessment  Metrics  (ANAM)  software  to  conduct  the  required  baseline  testing  and  continues  to 
use  ANAM4.4  DoD  had  sold  the  ANAM  software  license  to  Oklahoma  State  University  and, 
until  2014,  was  paying  an  annual  licensing  fee  for  up  to  400,000  ANAM4  traumatic  brain  injury 
(TBI)  assessment  battery  tests.5  DoD  also  contracted  with  the  Eyak  Corporation  to  provide 
trained  contract  staff  to  administer  the  ANAM4  tests  as  needed.  Prior  to  its  adoption  as  the 
NCAT  to  conduct  pre-deployment  baselines  in  2008,  the  ANAM  was  used  primarily  for  research 
on  the  effects  of  medication  and  other  chemicals  on  neurocognitive  function.10  Work  on 
developing  a  neurocognitive  ANAM  assessment  for  mTBI  began  in  2001,  but  use  of  the  mTBI 
battery  was  not  widespread.10'50  However,  as  a  result  of  the  new  requirement  for  baseline  testing 
on  all  deploying  Service  members  and  the  associated  licensing  and  contracting  expenses,  the 
overall  cost  of  the  ANAM  program  increased  significantly.  The  overall  cost  continued  to 
increase  through  2011  and  then  gradually  decreased  starting  in  2012  as  the  total  number  of 
deployments  decreased.21  As  summarized  in  Table  2,  DoD  spent  approximately  $47,723,144  to 
conduct  ANAM4  testing  between  Fiscal  Year  (FY)  2009  and  the  2nd  quarter  of  FY  2015. 


Table  2.  Total  Number  of  ANAM4  Tests  and  Associated  Costs,  FY  2009  -  2nd  Quarter  of  FY 
2015s 


FY 

Number  of  Tests 
Administered* 

Licensing  Fee 

Contract  Costs 

FY  2009 

296,272 

$2,000,000 

$11,506,169.01 

(FY  2009-2010) 

FY  2010 

307,616 

$2,000,000 

FY  2011 

325,037 

$2,000,000 

$7,192,749.27 

FY  2012 

255,134 

$2,000,000 

$4,897,378.00 

FY  2013 

246,619 

$2,000,000 

$4,622,299.70 

FY  2014 

196,368 

$2,000,000 

$4,410,268.31 

FY  2015 
(Quarters 
1+2) 

78,023 

$1,000,000  for  Q1+Q2 

$2,094,279.67  for  Q1+Q2** 

Totals: 

1,705,069 

$13,000,000 

$34,723,146.96 

Total  Cost  (Licensing  fees  + 
contract  costs): 

$47,723,143.96 

*  Total  number  of  tests  includes  reported  baselines,  post-injury,  clinical,  post-deployment,  and  unspecified  tests. 
**Contract  cost  listed  is  one  half  of  the  annual  contract  cost  for  FY  2015. 

Adapted  from  ANAM  Program  Office,  2015. 


The  current  licensing  and  contracting  model  for  the  ANAM4  program  remains  similar  to  the 
2008  model.  DoD  pays  a  licensing  fee  to  Oklahoma  State  University;  however,  instead  of 
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paying  $2  million  a  year  for  up  to  400,000  test  administrations,  DoD  now  pays  $1  million  for  up 
to  100,000  tests  per  year,  with  the  option  of  administering  more  tests  for  a  fee  of  $140,000  per 
20,000  additional  tests.  DoD  continues  to  contract  with  Eyak  Corporation  to  administer  on-site 
ANAM4  testing.  When  the  total  cost  of  the  program  and  the  total  number  of  tests  administered 
between  FY  2009  and  FY  2014  are  taken  into  account,  the  estimated  average  cost  of  each  test  is 
$26.81. 


Table  3.  Total  Number  of  Tests  and  Costs  per  Year5 


Fiscal 

Year 

Total  Tests 
Administered 

Contract  Cost 

License 

Fee 

Average  Cost 
Per  Test  Per 
Year*** 

FY  2009 

296,272 

$11,506,169.01* 

$2,000,000 

$26.58 

FY  2010 

307,616 

$2,000,000 

FY  2011 

325,037 

$7,192,749.27 

$2,000,000 

FY  2012 

255,134 

$4,897,378.00 

$2,000,000 

$27.03 

FY  2013 

246,619 

$4,622,299.70 

$2,000,000 

$26.85 

FY  2014 

196,368 

$4,410,268.31 

$1,000,000 

$27.55 

FY  2015 
(Quarters 
1+2) 

78,023 

$2,094,279.67** 

$1,000,000 

$39.66 

*  Hardware  purchases  accounted  for  a  larger  proportion  of  contract  costs  in  FYs  2009-2011. 

**Contract  cost  listed  is  one  half  the  annual  contract  cost  for  FY  2015. 

***These  costs  should  be  considered  approximate  estimates  based  on  the  information  provided. 

Adapted  from  AN  AM  Program  Office,  2015. 

According  to  data  obtained  from  the  ANAM  program  office,  a  total  of  1,841,489  tests  have  been 
administered  as  of  the  second  quarter  of  FY  2015  (Table  4).  Of  that  number,  1,627,202  were 
pre-deployment  baselines  and  84,524  tests  were  administered  for  post-deployment,  clinical, 
injury,  or  recovery  TBI  assessments.  An  additional  30,751  tests  were  administered  for 
unspecified  reasons. 


Table  4. 

Breakdown  of  Total  Number  of  ANAM4  Tests  by 

Reason 

Service 

Pre- 

Deployment 

Post- 

Deployment 

Clinical 

Injury 

Recovery 

Unspecified 

Total 

Army 

1,049,441 

52,174 

11,490 

2,301 

241 

20,723 

1,136,370 

Air 

Force 

303,738 

1,646 

1,415 

207 

17 

4,564 

311,587 

Marine 

Corps 

247,050 

824 

9,921 

2,054 

331 

2,587 

262,767 

Navy 

80,229 

522 

834 

135 

22 

911 

82,653 

Coast 

Guard 

2,195 

1 

4 

0 

0 

7 

2,207 

Non- 

Military 

43,549 

240 

120 

24 

1 

1,959 

45,893 

TOTAL 

1,726,202 

55,407 

23,784 

4,721 

612 

30,751 

1,841,477 

Adapted  from  ANAM  Program  Office,  2015. 
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As  shown  in  Table  5,  baselines  comprise  93.7  percent  of  total  tests  administered.  Post¬ 
deployment,  clinical,  injury  and  recovery  assessments  comprise  4.6  percent  of  the  total,  and  tests 
conducted  for  unspecified  reasons  comprise  1 .7  percent.  The  relatively  low  number  of  tests 
administered  for  post-deployment,  clinical,  injury,  and  recovery  assessments  in  the  management 
of  mTBI  compared  to  the  reported  incidence  of  mTBI  may  be  related  to  a  number  of  factors. 
Military  clinicians  do  not  always  use  ANAM4  in  their  assessment  and  management  of  mild 
traumatic  brain  injury  (mTBI)  since  the  ANAM  testing  policy  applies  only  to  mTBIs  that  occur 
during  deployment,4  and  approximately  80  percent  of  TBIs  occur  in  garrison.74  Clinicians  who 
specialize  in  caring  for  mTBI  patients  in  the  non-deployed  setting  may  have  access  to  a  more 
sophisticated  set  of  neuropsychological  and  physical  assessments  that  may  not  be  available  in 
deployed  settings.  Another  factor  may  be  that  earlier  in  the  program,  baseline  ANAM4  scores 
were  not  perceived  to  be  readily  accessible,  making  it  difficult  to  compare  post-injury  test  results 
to  a  Service  member’s  baseline.  Finally,  even  in  the  deployed  environment,  the  severity  of  the 
TBI  may  affect  the  use  of  ANAM4  in  assessing  the  injury.  In  cases  of  very  mild  TBI  that  occur 
in  theater,  clinicians  may  not  refer  Service  members  for  further  testing  with  ANAM4  if  they  do 
not  report  any  symptoms  and  they  do  not  demonstrate  any  impairment  on  the  Military  Acute 
Concussion  Evaluation  (MACE),  an  initial  screening  conducted  after  a  concussive  event  (or 
high-risk  exposure)  to  assess  whether  a  Service  member  has  sustained  an  mTBI.  A  full 
breakdown  of  the  ANAM4  test  reasons  by  date  of  test  administration  can  be  found  in  Appendix 
D. 


Table  5.  Breakdown  of  Test  Reason  by  Percentage  of  Total  Number  of  Tests  Administered5 


Service 

Pre- 

Deployment 

Post- 

Deployment 

Clinical 

Injury 

Recovery 

Unspecified 

Total 

Army 

92.3% 

4.6% 

1.0% 

0.2% 

0.0% 

1.8% 

100% 

Air  Force 

97.5% 

0.5% 

0.5% 

0.1% 

0.0% 

1.5% 

100% 

Marine 

Corps 

94.0% 

0.3% 

3.8% 

0.8% 

0.1% 

1.0% 

100% 

Navy 

97.1% 

0.6% 

1.0% 

0.2% 

0.0% 

1.1% 

100% 

Coast 

Guard 

99.5% 

0.0% 

0.2% 

0.0% 

0.0% 

0.3% 

100% 

Non- 

Military 

94.9% 

0.5% 

0.3% 

0.1% 

0.0% 

4.3% 

100% 

TOTAL 

93.7% 

3.0% 

1.3% 

0.3% 

0.0% 

1.7% 

100% 

Adapted  from  ANAM  Program  Office,  2015. 


5.2  Cost  Associated  with  Maintenance  of  a  Normative  Database 

One  of  the  primary  questions  asked  of  the  Defense  Health  Board  was  whether  a  normative 
database  could  be  used  in  lieu  of  continuing  baseline  testing.  A  preliminary  analysis  of  the 
current  ANAM4  normative  database,  stratified  only  by  sex  and  age  (in  one  year  increments), 
showed  a  significant  error  rate  in  classifying  scores  which  were  greater  than  one  standard 
deviation  above  or  below  the  mean.  For  asymptomatic  individuals  with  no  self-reported  history 
of  TBI  in  this  group,  analysis  using  the  normative  dataset  yielded  a  false  positive  rate  of  18.0 
percent,  a  false  negative  rate  of  46.3  percent,  and  an  overall  error  rate  of  64.3  percent.5 
Development  of  an  improved  ANAM4  normative  database  is  currently  in  progress. 
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Presuming  that  use  of  an  optimally  stratified  normative  database  could  provide  performance 
metrics  similar  to  those  found  using  baseline  comparison  testing,  routine  baseline  testing  might 
be  discontinued.  However,  the  normative  database  would  still  need  to  be  updated  periodically 
with  additional  baseline  tests  as  the  demographic  profile  of  the  military  changes  over  time. 
Additional  research  is  needed  to  determine  how  many  and  how  frequently  additional  baselines 
would  need  to  be  obtained  to  keep  the  normative  database  current.  DoD  would  also  need  to 
retain  the  infrastructure  necessary  to  conduct  post-injury  and  post-deployment  ANAM4  testing 
to  assess  the  cognitive  function  of  Service  members  who  sustain  mTBI  during  deployment. 

Since  this  post-injury/post-deployment  testing  infrastructure  would  need  to  remain  in  place,  it 
remains  unknown  whether  there  would  be  a  significant  cost  savings  on  licensing  or  contracting 
fees  compared  to  the  current  model. 

As  discussed  in  Section  3,  using  normative  data  in  lieu  of  individual  baselines  in  the  assessment 
of  mTBI  may  increase  the  possibility  of  false  negative  and  false  positive  classifications.  Service 
members  whose  cognitive  function  normally  falls  significantly  below  or  above  the  average  are 
especially  at  risk  for  post-mTBI  misclassification,  as  lower  functioning  individuals  are  more 
likely  to  register  as  false  positives  and  higher  functioning  individuals  are  more  likely  to  appear  as 
false  negatives.  These  false  indications  could  potentially  have  significant  impact;  Service 
members  who  test  below  the  norm,  even  though  that  reflects  their  true  baseline,  may  be 
subjected  to  unnecessary  and  expensive  testing,  may  be  inappropriately  prevented  from  returning 
to  duty,  and  may  be  incorrectly  deemed  to  have  a  Service-connected  disability.  High  functioning 
Service  members  who  fall  within  the  normal  range  on  post  mTBI  testing,  even  though  they  have 
a  persistent  mTBI-related  cognitive  deficit,  may  not  receive  appropriate  management,  may  be 
returned  to  duty  prematurely,  and  if  the  deficit  is  permanent,  may  not  be  recognized  as  having  a 
Service-connected  disability. 

5.3  Expanding  Testing  Beyond  the  Deployment  Cycle 

The  larger  population  in  garrison,  combined  with  injuries  from  training,  motor  vehicle  accidents, 
sports,  and  other  recreational  activities  results  in  a  larger  number  of  mTBIs.  Nearly  80  percent 
of  all  TBIs  occur  in  garrison,  and  mTBIs  comprise  approximately  80  percent  of  the  total  number 
of  TBIs  that  occur  in  Service  members.  Clinicians  are  not  currently  required  to  administer  the 
ANAM4  as  part  of  an  assessment  of  TBI  that  occurs  in  garrison,  but  the  testing  is  available  in 
many  locations,  and  the  test  battery  has  been  approved  as  an  optional  evaluation  in  a  new 
garrison  TBI  management  policy.76  If  research  indicates  that  ANAM4  is  a  valuable  component 
of  the  assessment  and  management  of  mTBI,  expanding  ANAM4  testing  beyond  the  deployment 
cycle  may  be  beneficial.  There  are  many  factors  that  may  affect  the  overall  cost  of  expanding 
the  program  to  non-deploying  personnel,  such  as  the  size  of  the  population  that  should  undergo 
baseline  testing.  For  example,  baselines  could  be  conducted  on  all  personnel,  only  on  those  who 
have  occupations  or  participate  in  activities  considered  high  risk  for  mTBI,  or  routine  baseline 
testing  could  be  discontinued.  If  the  approach  of  conducting  baselines  on  all  Service  members  or 
a  high-risk  subset  were  adopted,  the  resulting  increase  in  testing  would  likely  increase  the 
software-licensing  fee  and  possibly  the  contract  expenses  for  test  administration.  Even  if 
expanded  baseline  testing  was  not  planned,  some  level  of  additional  baseline  testing  may  be 
needed  to  ensure  the  normative  database  is  regularly  updated  and  reflects  the  demographic 
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characteristics  of  the  garrison  population,  which  are  likely  different  from  those  of  the  deployed 
population. 

Finding  5:  Currently,  there  is  insufficient  evidence  to  support  expanding  the  use  of 
ANAM4  administration,  to  include  periodic  baseline  assessments  of  cognitive  function, 
outside  of  the  deployment  cycle. 

Recommendation  5:  DoD  should  defer  a  decision  to  expand  ANAM4  baseline  testing 
outside  of  the  deployment  cycle  until  additional  research  demonstrates  that  baseline 
testing  improves  the  evaluation  and  management  of  mild  traumatic  brain  injury. 

5.4  Options  to  Reduce  Costs 

Recent  cost  savings  in  the  ANAM  testing  program  have  occurred  primarily  due  to  restructuring 
of  the  software-licensing  fee  and  decreased  demand  for  testing  with  fewer  deploying  personnel. 
Assuming  there  is  no  change  in  the  current  neurocognitive  testing  policy,  annual  costs  would 
likely  remain  stable  unless  there  was  a  significant  increase  in  the  number  of  personnel  deploying. 

One  approach  to  reduce  or  limit  costs  would  be  to  limit  individual  baseline  testing  to  sub¬ 
populations  that  are  at  higher  risk  of  TBI  while  deployed  due  to  their  military  occupational 
specialty  or  deployment  duties.  However,  personnel  who  do  not  receive  baseline  testing  may 
perceive  they  are  not  being  provided  an  appropriate  opportunity  to  accurately  document  their 
baseline  cognitive  function  in  the  event  they  experience  an  mTBI.  Considering  the  cost 
necessary  to  maintain  a  minimum  ANAM4  testing  infrastructure,  it  is  not  clear  how  significant 
the  cost  savings  may  be  with  this  approach. 

Overall,  questions  remain  regarding  the  utility  and  cost-effectiveness  of  ANAM4  testing  in 
managing  mTBI.  Whether  using  an  optimally  stratified  normative  database  would  be  as  accurate 
as  using  individual  baseline  test  results  to  identify  cognitive  deficits  following  mTBI  at  a  lower 
cost  remains  unclear. 

5.5  Costs  Associated  with  the  Immediate  Post-Concussion  Assessment  and 
Cognitive  Testing  Program 

The  Immediate  Post-Concussion  Assessment  and  Cognitive  Testing  Program  ImPACT  is  an 
Internet-based  test  currently  used  by  the  U.S.  Army  Special  Operations  Command  (USASOC)  to 
assess  for  cognitive  deficits  associated  with  mTBI  in  Service  members.  The  contract  for  FY 
2015  has  associated  costs  of  $2  per  baseline  test  and  $8  per  post-injury  test.  There  is  an 
additional  option  to  train  four  medical  officers  for  $3,840  to  administer  and  interpret  the 
ImPACT  test.  The  contract  requires  that  the  test  provider  provide  24/7  access  to  the  online 
ImPACT  test  and  test  results,  including  all  previous  tests  taken  by  any  Service  member.  The  test 
providers  also  deliver  technical  support  as  needed.22  The  program  does  not  provide  for 
equipment  or  proctors  to  conduct  testing. 
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Finding  6:  DoD  is  currently  using  at  least  two  NCATs  (ANAM  and  ImPACT)  to  assess 
neurocognitive  function.  There  are  significant  differences  in  the  pricing  structure  for 
these  tools  and  studies  have  not  shown  either  tool  to  have  a  distinct  overall  advantage  in 
contributing  to  the  management  of  mild  traumatic  brain  injury. 

Recommendation  6:  DoD  should  conduct  a  competitive  bidding  process  to  select  the 
most  cost-effective  approach  to  meet  requirements  for  all  non-specialized  neurocognitive 
assessments  for  the  management  of  mild  traumatic  brain  injury. 
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6.  Areas  for  Future  Research 

Recent  publications  outline  the  need  for  additional  research  on  the  development  and  appropriate 
use  of  computerized  neurocognitive  assessment  tools  (NCATs). 13,16,24,25  These  tools  have 
become  widely  used  in  the  assessment  and  management  of  mild  traumatic  brain  injury  (mTBI); 
however,  challenges  remain  in  improving  their  sensitivity,  accuracy,  and  the  methodology  for 
interpreting  test  results.  Based  on  a  review  of  the  current  ANAM4  program  and  available 
literature  and  expert  opinion,  the  Subcommittee  has  identified  several  focus  areas  for  future 
research,  as  outlined  below. 

6. 1  ANAM4  Program  Cost-Effectiveness 

There  is  no  apparent  consensus  on  the  best  approach  to  measure  cost-effectiveness  of 
computerized  NCATs  and  no  published  research  was  located  assessing  the  cost-effectiveness  of 
the  ANAM4  program.  As  discussed  in  Section  5,  DoD  was  recently  able  to  reduce  the  cost  of 
the  overall  program  due  to  the  decreased  demand  for  testing  with  fewer  personnel  deploying. 

The  effectiveness  of  the  ANAM4  in  contributing  to  the  management  of  mTBI,  including 
decisions  on  return  to  duty,  has  not  been  adequately  assessed.  Some  clinicians  who  are  using  or 
had  used  the  ANAM4  endorsed  the  value  of  the  test  in  providing  useful  objective  data  to  assess 
cognitive  function  following  an  mTBI  or  high-risk  exposure.  They  indicated  the  information 
was  useful  in  making  decisions  on  returning  Service  members  to  duty  or  referring  them  for 
additional  evaluation.  Additional  work  on  defining  and  capturing  the  information  necessary  to 
determine  overall  cost  effectiveness  is  needed.  The  1.8  million  ANAM4  baselines  collected  by 
DoD  will  provide  valuable  data  for  this  work. 

6.2  Accuracy  of  Using  Baseline  versus  Normative  Values  in  Assessing 

mTBI 

As  discussed  in  Section  3,  there  is  general  consensus  that  comparing  post-mTBI  NCAT  scores  to 
valid  pre-injury  baselines  is,  in  theory,  more  accurate  than  comparing  post-injury  scores  to 
normative  values.16  However,  on  a  population  level,  studies  have  not  clearly  or  consistently 
demonstrated  this  advantage.  ’  ’  The  inherent  variation  in  the  quality  of  baselines  collected  via 
large-scale  computerized  testing  under  varying  conditions,  the  inherent  temporal  variation  in 
individual  human  cognitive  function  with  physical  or  psychological  stressors,  the  inherent 
limitations  in  the  current  generation  of  NCATs,  and  limitations  in  the  methods  for  analyzing  test 
results  all  likely  contribute  to  this  phenomenon.  DoD  is  currently  developing  a  larger  and  more 
stratified  ANAM4  normative  database.  Upon  completion,  a  retrospective  comparison  of  baseline 
and  normative  data  with  ANAM4  data  from  post  mTBI  clinical  evaluations  should  provide 
important  information  on  the  relative  value  of  having  baseline  test  data.  As  stated  by  Iverson  et 
al,  this  research  will  help  determine  whether  having  individual  baselines  contributes  in  a 
meaningful  way  to  the  evaluation  and  management  of  mTBI  and  retum-to-duty  decisions  in  the 
context  of  a  multidimensional  clinical  assessment.16 


Future  Research 


33 


Defense  Health  Board 


6.3  Other  Neurocognitive  Assessment  Tools 

The  field  of  computerized  neurocognitive  testing  is  growing,  with  new  tests  and  technologies 
emerging  regularly.'  ’  As  technology  continues  to  evolve,  accessibility,  feasibility,  and 
accuracy  are  important  factors  to  consider  as  the  Department  of  Defense  (DoD)  reviews  new 
assessment  tools  to  evaluate  and  manage  mTBI.  Additional  research  into  the  psychometric 
properties  of  existing  NCATs  in  the  assessment  of  mTBI  is  needed,  particularly  with  respect  to 
blast-related  TBI.  '  These  studies  will  provide  data  that  will  help  DoD  determine  whether 
ANAM4  has  comparable  or  superior  reliability  and  clinical  utility  compared  to  other  NCAT 
batteries. 

DoD  briefly  funded  development  and  testing  of  a  new  neurocognitive  assessment  tool,  the 

Defense  Automated  Neurobehavioral  Assessment  (DANA),  as  a  hand-held  device  to  assess 

mTBI  in  military  populations.  Unlike  ANAM4,  DANA  may  be  administered  on  a  smartphone, 

8 1 

making  the  test  more  easily  accessible  to  evaluate  injured  Service  members  in  the  field. 

DANA  was  cleared  by  the  FDA  as  a  medical  device  in  October  2014,  however,  at  the  time  of 
this  report,  DoD  is  no  longer  funding  additional  development  of  this  device. 

It  is  important  to  remember  that  NCATs  are  only  one  component  of  a  comprehensive  clinical 
evaluation  of  mTBI.  In  light  of  this,  DoD  is  pursuing  research  on  new  assessments  such  as  brain 
imaging,  postural  stability,  vision  testing,  biomarkers,  and  other  physiological  evaluations.  The 
2009  clinical  practice  guidelines  published  by  DoD  and  the  Department  of  Veterans  Affairs 
outline  the  benefits  of  a  multimodal  approach  to  the  evaluation  and  management  mTBI.  There 
is  increasing  evidence  that  multimodal  assessments  that  include  evaluation  of  physical 
characteristics,  such  as  balance  and  eye  tracking,  in  addition  to  neurocognitive  testing  may 
provide  a  more  effective  approach  to  the  evaluation  and  management  of  mTBI.14’26’27  New 
research  on  brain  imaging  and  biomarkers  continues  to  elucidate  the  physiological  signs  of 
mTBI,  with  breakthroughs  in  finding  hard-to-see  microbleeds  and  white  tissue  damage  '  ’  and 
identifying  biomarker  molecules  associated  with  injured  and  recovering  brain  tissue.85  A 
handful  of  biomarkers  associated  with  moderate  to  severe  TBI  have  been  identified,  and  research 
is  ongoing  to  determine  whether  they  are  also  present  and  provide  valuable  clinical  insights  in 

oc 

mTBI.  While  still  a  new  field,  these  tests  offer  an  objective  assessment  of  a  Service  member’s 
brain  health  following  an  injury  that  could  contribute  to  a  more  comprehensive  post-injury 
evaluation. 

Finding  7:  There  is  evidence  from  academic  research  and  from  DoD  that  multimodal 
approaches  including  imaging,  use  of  biomarkers,  and  physical  diagnostic  techniques 
may  be  more  effective  than  NCATs  alone  in  evaluating  and  managing  mild  traumatic 
brain  injury. 

Recommendation  7:  DoD  should  sustain  and  advance  research  to  determine  if  a 
multimodal  approach  can  be  developed  that  is  cost-effective  and  superior  to  NCAT 
testing  alone  in  assessing  and  managing  mild  traumatic  brain  injury. 
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6.4  Test  Reliability  of  ANAM4  and  Its  Individual  Subtests 

Given  the  normal  fluctuation  that  occurs  in  the  cognitive  functions  measured  by  ANAM4  and  the 
relatively  new  application  of  AN  AM  to  mTBI  evaluation,  there  is  need  for  research  on  the 
impact  of  depression,  deployment,  posttraumatic  stress,  acute  stress,  fatigue,  and  pain  on  subtest 
and  overall  test  scores.  This  research  should  be  conducted  specifically  in  the  military  population, 
as  Service  members  face  unique  stressors  and  experiences.  This  research  may  also  yield  new 
methods  to  evaluate  the  mental  health  of  Service  members  not  necessarily  related  to  mTBI, 
allowing  for  improved  diagnosis,  documentation,  and  management  of  mental  illness  and  post- 

64 

traumatic  stress. 

The  heterogeneity  of  mTBI  makes  diagnosis  and  management  challenging.  The  differences 
between  blunt  trauma-related  mTBI  and  blast-related  mTBI  merit  further  research.1 1  Currently, 
data  suggest  that  there  is  little  difference  in  the  recovery  process  between  the  two  injuries  when 
evaluated  using  NCATs.  '  The  large  number  of  blast-related  mTBI  in  recent  conflicts 
highlights  the  need  for  state-of-the-art  techniques  to  both  assess  and  manage  this  condition  as 
well  as  provide  more  effective  treatment.  Refinement  of  mTBI  assessment  tools  such  as 
ANAM4  and  other  clinical  exams  may  allow  clinicians  to  better  understand  the  physiology  of 
mTBI  and  monitor  recovery.  Identification  of  sensitive  and  specific  biomarkers  for  TBI  with 
diagnostic  and  prognostic  properties  would  also  be  of  significant  value. 

Another  area  of  interest  for  the  Subcommittee  is  practice  effects.  While  practice  effects  for 
repeat  administrations  of  ANAM4  have  been  demonstrated  in  healthy  individuals,  it  appears  that 

25  28  86 

it  is  also  present  in  individuals  who  have  experienced  mTBI,  albeit  at  a  lesser  magnitude.  ’  ’ 

As  researchers  learn  more  about  which  cognitive  functions,  such  as  simple  reaction  time  (SRT), 
tend  to  demonstrate  deficits  because  of  mTBI,  it  will  be  important  to  understand  how  practice 
effects  influence  scores  on  these  subtests  after  serial  administrations  of  ANAM4.  Practice 
effects  tend  to  improve  the  scores  of  Service  members  who  have  taken  the  test  multiple  times.  If 
injured  Service  members’  test  scores  improve  after  they  have  taken  ANAM4  multiple  times  post¬ 
injury,  the  improvement  may  be  a  result  of  practice  effects,  rather  than  full  recovery.  This  may 
result  in  an  individual  being  returned  to  duty  prematurely,  putting  them  at  risk  for  additional 
TBIs  when  their  brains  have  not  fully  recovered.  There  is  some  evidence  that  suggests  practice 
effects  are  diminished  in  individuals  who  have  experienced  mTBI.  If  this  is  the  case, 
manifestation  of  increased  practice  effects  or  improved  scores  may  indicate  recovery. 

6.5  Research  on  Symptomatology  of  and  Recovery  from  mTBI 

Most  individuals  who  experience  mTBI  recover  within  a  few  weeks;  however,  some  individuals 
experience  persistent  symptoms  or  signs  of  mTBI.  It  is  currently  difficult  to  accurately  correlate 
persistent  symptoms  with  performance  on  ANAM4,  as  the  test  may  not  be  able  to  discriminate 
between  performance  decrements  related  to  chronic  effects  of  mTBI  or  malingering.16 
Conversely,  months  after  their  injury  individuals  may  report  no  symptoms  but  continue  to  score 
below  their  baseline  on  ANAM4.  In  these  situations,  comprehensive  neuropsychological  testing 
may  be  indicated  to  more  precisely  discern  the  potential  roles  of  residual  cognitive, 
psychological,  and  physical  factors  affecting  ANAM4  test  scores.  In  general,  continued 
development  and  refinement  of  better  tools  and  methods  to  discriminate  between  performance 
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decrements  related  to  mTBI  and  other  causes  will  allow  clinicians  to  better  manage  patients  by 
targeting  interventions  to  the  appropriate  underlying  causes. 

Finding  8:  Individuals  with  persistent  symptoms  following  mild  traumatic  brain  injury 
are  often  found  to  have  comorbidities  such  as  post-traumatic  stress,  depression,  sleep 
disturbances,  pain,  and  anxiety.  There  is  insufficient  research  on  the  impact  of  these 
comorbidities  on  ANAM4  test  performance. 

Recommendation  8:  DoD  should  conduct  additional  research  to  determine  the  effects  of 
comorbidities  on  ANAM4  test  performance. 

With  the  conversion  of  more  than  1.8  million  ANAM4  test  scores  into  an  organized  database, 
DoD  has  the  opportunity  to  conduct  extensive  retrospective  and  prospective  research.  These 
data,  along  with  additional  patient  outcome  information,  provide  valuable  longitudinal  data  on 
thousands  of  Service  members.  Analysis  of  these  data  in  conjunction  with  available  medical 
histories  and  symptom  reports  will  allow  researchers  to  investigate  the  impact  of  multiple 
mTBIs,  long-term  recovery,  the  effects  of  repeat  testing,  correlation  of  symptom  resolution  with 
test  scores,  comorbidities,  lingering  symptoms,  and  persistent  low  scores  on  ANAM.  If  a 
deidentified  version  of  the  data  was  made  readily  available  to  researchers,  it  would  provide  a 
valuable  and  cost-effective  resource  to  leverage  both  civilian  and  military  expertise  to  more 
rapidly  advance  the  science  of  using  NCATs  in  evaluating  and  managing  mTBI. 

Finding  9:  DoD  has  collected  more  than  1.8  million  baseline,  clinical,  and  other 
ANAM4  test  results. 

Recommendation  9:  DoD  should  make  a  deidentified  version  of  these  data  available  to 
civilian  researchers  to  leverage  those  resources  in  accomplishing  additional  analyses 


6.6  ANAM4  Test  Score  Interpretation  Methodology 

Although  ANAM’s  technology  allows  for  precise  measurements  of  reaction  time  and  other 
psychometric  measures,  interpretation  of  ANAM4  score  results  remains  inconsistent.  Methods 
for  calculating  reliable  change,  as  well  as  interpreting  change  in  score  between  baseline  and  post- 
injury  test,  vary  widely.  In  addition,  the  methodologies  for  comparing  post-injury  ANAM4 
scores  to  baseline  and  normative  data  vary,  as  clinicians  may  set  different  definitions  of 
impairment.  For  example,  when  comparing  post-injury  ANAM4  scores  to  a  normative  dataset,  a 
clinician  may  classify  a  Service  member  as  “impaired”  if  they  score  one  standard  deviation 
below  the  norm  on  one  subtest,  while  others  may  classify  a  Service  member  as  impaired  if  they 
score  two  standard  deviations  below  the  mean.5  These  differences  lead  to  challenges  in 
calculating  the  sensitivity  and  specificity  of  using  individual  baselines  versus  normative  data, 
making  accurate  comparison  of  the  two  approaches  difficult.  Key  areas  for  research,  such  as 
investigation  of  the  best  methodology  for  interpreting  post-injury  scores  when  pre-deployment 
baselines  are  available  and  identification  of  the  best  algorithms  for  interpreting  post-injury  scores 
when  comparing  to  normative  data,  will  aid  in  that  endeavor.  Researchers  have  also  suggested 
that  DoD  “develop  and  evaluate  clinical  algorithms,  with  known  false  positive  rates,  for 
identifying  and  quantifying  cognitive  impairment  following  mTBI.”16 
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When  investigating  practice  effect,  it  will  also  be  important  to  review  which  method  is  most 
accurate  for  interpreting  changes  in  score  on  ANAM4  retests.  Two  methodologies  that  are 
currently  used  are  the  reliable  change  method  and  the  regression-based  change  model.  A  head- 
to-head  comparison  of  the  two  methods  would  help  determine  the  impact  of  practice  effects  and 
assess  the  reliability  of  ANAM4.  While  these  methods  are  currently  widely  used,  some 
researchers  argue  that  more  sophisticated  methods  for  interpreting  change  on  NCATs  should  be 
developed.  These  may  include  refining  the  reliable  change  methodology  to  correct  for  practice 
effects  and  stratification  of  confidence  intervals  for  change  based  on  level  of  baseline 
performance.16  Researchers  have  also  suggested  applying  multivariate  base  rate  analyses  to  the 
reliable  change  methodology  to  allow  for  quantification  of  the  likelihood  of  showing  one  reliable 
change  on  one  or  more  scores  when  multiple  scores  are  considered  simultaneously. 

6.7  Test  Environment  and  Administration 

Testing  environment  has  been  identified  as  a  factor  potentially  contributing  to  variation  in  NCAT 
testing  results.  Guidelines  for  administering  ANAM4  are  outlined  in  the  ANAM4  User  Manual, 
and  clearly  state  that  the  testing  environment  be  clean,  quiet,  comfortable,  and  free  of  distraction. 
The  current  policy  of  administering  pre-deployment  baseline  testing  in  a  group  setting  may  not 
adhere  to  these  standards,  possibly  resulting  in  Service  members  scoring  lower  when  ANAM4  is 
administered  that  way.  This  belief  is  supported  by  the  research  of  Moser,  et  al,  who  found  that 
athletes  score  significantly  lower  on  computerized  neurocognitive  tests  when  tests  are 
administered  in  a  group  setting  rather  than  individually.”  Further  study  of  the  psychometric 
implications  of  completing  the  ANAM4  in  group  versus  individual  settings  will  contribute  to 
determining  the  best  approach  to  obtain  more  consistent  and  accurate  baselines. 

Other  concerns  include  the  effect  of  confusion  or  misunderstanding  about  the  test  or  testing 
instructions  on  test  scores.  It  is  unclear  whether  low  scores  that  occur  because  of  confusion  are 
highlighted  by  validity  indicators  or  not.60  If  these  low  scores  are  not  identified  by  the 
algorithms  used  to  assess  validity,  it  is  possible  that  Service  members  who  were  confused  by 
testing  instructions  will  demonstrate  lower  ANAM4  baseline  scores,  thereby  reducing  the 
accuracy  of  baseline  comparison  to  post-injury  test  scores  if  they  sustain  an  mTBI. 

Finding  10:  DoD  currently  administers  ANAM4  baseline  tests  in  group  settings; 
however,  research  indicates  that  administering  computerized  neurocognitive  tests  in  a 
group  setting  may  affect  an  individual’s  test  score.28 

Recommendation  10:  DoD  should  determine  whether  and  to  what  extent  the  group 
testing  environment  affects  ANAM4  baseline  tests  scores  as  compared  to  individual 
ANAM4  testing. 
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7.  Appendices 

Appendix  A:  Letter  of  Request 


UNDER  SECRETARY  OF  DEFENSE 

4000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-4000 

JUL  2  5  2014 

PERSONNEL  AND 
READINESS 

MEMORANDUM  FOR  PRESIDENT,  DEFENSE  HEALTH  BOARD 

SUBJECT:  Request  to  Review  the  Scientific  Evidence  of  Using  Population  Normative  Values 
for  Post-Concussive  Computerized  Neurocognitive  Assessments 

As  post-concussive  neurocognitive  assessment  continues  to  evolve,  the  scientific, 
military,  and  sports  communities  have  raised  questions  regarding  the  utility  of  using  baseline 
assessments  for  post-injury  comparison,  versus  using  population  normative  values,  to  define 
injury-related  cognitive  deficits  and  help  inform  retum-to-duty  and  return-to-play  decisions.  The 
Department  currently  collects  computerized,  pre-deployment  baseline  neurocognitive  assessment 
data  using  the  Automated  Neuropsychological  Assessment  Metrics  4  (ANAM4).  Baseline 
ANAM4  data  are  available  for  retrieval  and  comparison  to  post-injury  ANAM4  assessment 
results  should  a  Service  member  experience  a  concussive  injury.  Since  July  2007,  the 
Department  completed  more  than  1.5  million  pre-deployment  baseline  tests  and  executed  26,524 
post-injury  comparison  assessments. 

The  Military  Services  have  raised  concerns  about  the  utility  and  logistics  of  continuing  to 
collect  pre-deployment  baseline  neurocognitive  tests  because  emerging  scientific  evidence 
suggests  that  before  and  after  comparative  testing  using  baselines  may  be  no  more  effective  than 
using  relevant  population  normative  values  for  the  detection  of  cognitive  deficits  associated  with 
the  concussion.  Therefore,  1  request  that  the  Defense  Health  Board  (DHB)  examine  the  state-of- 
the-science  on  neurocognitive  assessment  testing,  and  recommend  the  DHB  evaluation  consider 
the  following  questions: 

1 )  Does  the  current  state-of-the  science  demonstrate  a  continued  need  for  baseline 
computerized  neurocognitive  tests  to  make  retum-to-duty/play  determinations? 

2)  Is  the  current  dataset  of  military  relevant  normative  values  for  the  AN  AM4  (sample 
size  107,000)  an  adequately  sized  population  to  generate  age,  gender,  education, 
and  rank-matched  military  normative  values,  or  should  a  larger  dataset  be 
implemented  for  the  norms? 

3)  Are  population  normative  values  (assuming  an  adequate  number  and 
military-relevant  demographic  profile)  as  scientifically  sound  as  pre-deployment 
baseline  tests  for  reliably  detecting  post-concussive  neurocognitive  deficits  (within 
the  limitations  of  ANAM4)  for  retum-to-duty  decision  making  and  prognosis? 

4)  Is  there  any  utility  to  expanding  the  use  of  neurocognitive  assessment  testing  of 
military  populations  beyond  the  deployment  cycle  (pre-deployment,  post-injury, 
post-deployment)? 
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5)  Is  any  additional  direction  for  future  research  in  neurocognitive  assessment  testing 
needed  to  improve  protection  of  the  fighting  force? 

6)  What  is  the  cost  benefit  of  performing  baseline  testing  for  the  Military  Services  in  a 
fiscally  constrained  environment,  when  logistics,  contracts,  personnel,  and 
equipment  sustainment  are  taken  into  consideration? 

The  point  of  contact  for  this  action  is  Ms.  Elizabeth  Fudge,  Defense  Health  Agency, 
Health  Care  Operations  Directorate.  Ms.  Fudge  may  be  reached  at  (703)  681-8295  or 
elizabeth.fudge@dha.mil. 
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Appendix  B:  Terms  of  Reference 

Defense  Health  Board 

Review  of  Scientific  Evidence  of  Using  Population  Normative  Values  for  Post-Concussive 

Neurocognitive  Assessments 

TERMS  OF  REFERENCE 


These  terms  of  reference  establish  the  objectives  for  the  Defense  Health  Board’s  (DHB) 
examination  of  the  state  of  the  science  in  neurocognitive  assessment  testing  in  evaluating  mild 
traumatic  brain  injury  (tnTBI,  or  concussion)  and  the  effectiveness  of  the  Department  of  Defense’s 
(DoD’s)  current  testing  program.  The  terms  outline  the  scope  of  the  Board’s  examination  as  well 
as  the  Board’s  methodology  for  responding  to  DoD’s  request. 

Mission  Statement:  The  DHB  will  conduct  a  comprehensive  review  of  the  Automated 
Neuropsychological  Assessment  Metrics  4  (ANAM4)  neurocognitive  assessment  testing  in  the 
evaluation  of  mTBI  and  offer  recommendations  regarding  the  appropriateness  of  using 
normative  values,  as  opposed  to  individual  baseline  tests,  when  making  return-to-duty  decisions; 
the  adequacy  of  the  current  normative  dataset;  the  use  of  neurocognitive  testing  beyond  the 
normal  deployment  cycle;  and  future  research. 

Issue  Statement;  On  July  25,  2014,  the  Under  Secretary  of  Defense  for  Personnel  and 
Readiness  requested  the  DHB  review  the  scientific  evidence  of  using  population  normative 
values  for  post-concussive  computerized  neurocognitive  testing.  The  stated  premise  for  this 
request  was  that  emerging  scientific  evidence  suggests  that  using  individual  baseline 
neurocognitive  assessments  for  post-injury  comparison  may  be  no  more  effective  than  using 
population  normative  values  to  define  injury-related  cognitive  deficits  and  inform  return-to-duty 
and  retum-to-play  decisions.  Consequently,  the  Military  Services  have  expressed  concern  about 
the  utility  of  continuing  to  collect  pre-deployment  baseline  neurocognitive  test  data. 
Computerized,  pre-deployment  baseline  neurocognitive  assessment  testing  using  the  ANAM4  is 
currently  performed  on  Service  members  within  12  months  before  deployment.  Baseline 
ANAM4  data  are  available  for  retrieval  and  comparison  to  post-injury  ANAM4  assessment 
results  should  a  Service  member  experience  mTBI.  As  of  July  25,  2014,  the  Department 
completed  more  than  1.5  million  pre-deployment  baseline  tests  and  executed  26,524  post-injury 
comparison  assessments  since  July  2007. 

Objectives  and  Scope:  The  DHB  will  address  the  following  in  its  report: 

1.  Review  the  current  state  of  the  science  to  determine  whether  there  is  a  continued  need  for 
individual  baseline  neurocognitive  testing,  as  one  element  of  making  retum-to-duty/play 
determinations; 

2.  Determine  whether  the  current  dataset  of  military  relevant  normative  values  for  the  ANAM4 
is  adequately  sized  to  generate  age,  gender,  education  and  rank-matched  military  normative 
values; 
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3.  Determine  whether  population  normative  values,  as  one  element,  are  as  scientifically  sound 
as  pre-deployment  baseline  tests  for  reliably  detecting  post-concussive  neurocognitive 
deficits,  as  one  element  for  return-to-duty  decisions  and  the  value  of  ANAM4  in  making  a 
prognosis; 


4.  Determine  whether  expanding  the  use  of  computerized  ANAM4  neurocognitive  assessment 
testing  of  military  populations  in  addition  to  the  deployment  cycle  (pre-deployment,  post¬ 
injury,  post-deployment),  such  as  in  garrison,  is  of  value; 

5.  Identify  directions  for  future  research  in  computerized  ANAM4  neurocognitive  assessment 
testing  to  protect  the  fighting  force  and  to  improve  the  effectiveness  of  mTBI  assessment, 
treatment,  rehabilitation  and  disposition;  and 

6.  Compare  the  cost  and  benefits  of  performing  baseline  testing  in  the  Military  Services  when 
logistics,  contracts,  personnel,  and  equipment  sustainment  are  considered. 


Methodology:  The  Neurological/Behavioral  Health  Subcommittee  will  meet  regularly  to  review 
the  literature,  receive  briefings,  and  conduct  site  visits  as  needed.  Using  the  information  from 
these  research  activities,  the  Subcommittee  will  develop  findings  and  recommendations  based  on 
the  current  state  of  neuroscience.  The  members  will  present  their  preliminary  findings  and 
recommendations  to  the  DHB  for  consideration  and  deliberation.  The  DHB  will  deliberate  the 
findings,  during  which  time  members  may  propose  additional  recommendations,  and  vote  on 
these  collective  recommendations  in  an  open  public  session. 

Deliverable:  The  DHB  will  deliberate  the  final  findings  and  recommendations  presented  by  the 
Subcommittee  in  2015  and  produce  the  final  report  immediately  following  for  presentation  to  the 
Department.  The  Subcommittee  will  provide  progress  updates  to  the  Board  at  each  DHB 
meeting  before  then. 

Membership:  The  Neurological/Behavioral  Health  Subcommittee  members  will  conduct  the 
primary  investigation  and  will  consult  subject  matter  experts  as  needed. 


Support: 

1 .  The  DHB  office  will  provide  any  necessary  administrative,  analytical,  research,  and  logistical 
support  for  the  Subcommittee  and  Board. 

2.  Funding  for  this  review  is  included  in  the  DHB  operating  budget. 
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Appendix  C:  ANAM4  TBI  Test  Descriptions 

Demographics  Module 

The  demographics  module  allows  users  to  enter  a  wide  variety  of  information  including  name, 
age,  gender,  ethnicity,  medical  diagnosis,  medications,  and  additional  comments  that  the 
researcher  or  clinician  finds  useful. 

TBI  Questionnaire 

The  TBI  Questionnaire  is  designed  to  assess  injury  history  and  related  symptomatology. 

Sleep  Scale 

The  ANAM  sleepiness  scale  has  been  designed  to  provide  a  state  and/or  trait  assessment  of 
energy-fatigue  level.  This  test  permits  self-assessment  of  the  user’s  sleep/fatigue  state  (and/or 
trait).  The  user  is  presented  with  seven  different  statements  of  alertness/sleepiness  ranging  from 
“feeling  very  alert,  wide  awake,  and  energetic”  to  “Very  sleepy  and  cannot  stay  awake  much 
longer.”  The  user  is  instructed  to  select  the  one  statement  that  best  matches  the  current  state. 

Mood  Scale  II  -  Revised 

The  Moodscale2-R  is  designed  to  assess  either  mood  state  of  trait  in  participants  in  six 
subcategories  that  include  Vigor  (high  energy-level),  Happiness  (positive  disposition), 
Depression  (dysphoria),  Anger  (negative  disposition),  Fatigue  (low-energy  level),  and  Anxiety 
(anxiety  level). 

This  test  permits  self-  assessment  of  the  user’s  mood  state  in  seven  categories:  Vigor  (high 
energy-level),  Happiness  (positive  disposition),  Depression  (dysphoria),  Anger  (negative 
disposition),  Fatigue  (low-energy  level),  and  Anxiety  (anxiety  level),  and  a  new  subcategory  of 
Restlessness  (motor  agitation).  The  user  is  presented  with  a  scale  of  numbered  blocks  ranging 
from  0  to  6,  with  “0”  having  the  verbal  anchor  “Not  at  all,”  the  midpoint  “3”  labeled 
“Somewhat”  and  “6”  labeled  “Very  Much.”  The  user  is  presented  a  series  of  adjectives,  each 
adjective  contributing  to  one  of  the  mood  categories,  and  is  instructed  to  select  the  box/number 
that  best  represents  the  current  state  with  respect  to  the  presented  adjective. 

Simple  Reaction  Time 

The  results  of  this  test  are  used  as  an  index  of  visuo-motor  response  timing. 

This  test  measure  simple  reaction  time  by  presenting  the  user  with  a  series  of  “  *  ”  symbols  on 
the  display.  The  user  is  instructed  to  respond  as  quickly  as  possible  by  pressing  a  button  each 
time  the  stimulus  appears. 

Code  Substitution  -  Learning 

Results  of  this  test  are  used  as  an  index  of  visual  search,  sustained  attention,  and  encoding. 

In  this  test  the  user  must  compare  a  displayed  digit-symbol  pair  with  a  set  of  defined  digit- 
symbol  pairs,  or  the  key.  The  user  presses  the  designated  buttons  to  indicate  whether  the  pair  in 
question  represents  a  correct  or  incorrect  mapping  relative  to  the  key.  In  the  Learning  phase 
(simultaneous  presentation  mode),  the  defined  pairs  are  presented  on  the  screen  along  with  the 
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digit-symbol  pair  in  question.  In  the  Delayed  Memory  test  (to  follow  later  in  the  battery)  the 
comparison  stimuli  are  again  presented  individually  without  the  key. 

Procedural  Reaction  Time 

This  test  measures  the  reaction  time  and  processing  efficiency  associated  with  following  a  simple 
set  of  mapping  rules. 

There  are  three  possible  blocks  of  trials  for  this  test.  In  the  Basic  Block,  the  user  is  presented 
with  a  number  constructed  on  the  display  using  a  large  dot  matrix  (either  a  2,  3,  4,  or  5).  The 
user  is  instructed  to  press  on  designated  button  for  a  “low”  number  (2  or  3)  and  another 
designated  button  for  a  “high”  number  (4  or  5). 

Matching  to  Sample 

Results  of  this  test  are  used  as  an  index  of  spatial  processing  and  visuo-spatial  working  memory. 

During  this  test  the  user  views  a  pattern  produced  by  eight  shaded  cells  in  a  4  x  4  sample  grid. 
The  sample  is  then  removed  and  two  comparison  patterns  are  displayed  side  by  side.  One  grid  is 
identical  to  the  sample  grid  and  the  other  grid  differs  by  one  shaded  cell.  The  user  is  instructed 
to  press  a  designated  button  to  select  the  grid  that  matches  the  sample. 

Mathematical  Processing 

Results  of  this  test  are  used  as  an  index  of  basic  computational  skills,  concentration,  and  working 
memory. 

During  this  task,  an  arithmetic  problem  involving  three  single-digit  numbers  and  two  operators  is 
displayed  (e.g.,  “5-2  +  3  =”).  The  user  presses  buttons  to  indicate  whether  the  answer  to  the 
problem  is  less  than  five  or  greater  than  five. 

Code  Substitution  -  Delayed  (Recognition) 

Results  of  this  test  are  used  as  an  index  of  delayed  memory. 

In  this  test  the  user  must  compare  the  displayed  digit-symbol  pair  with  the  digit-symbol  pairs,  or 
key,  presented  during  the  Code  Substitution  -  Learning  test.  The  user  presses  designated  buttons 
to  indicate  whether  the  pair  in  question  represents  a  correct  or  incorrect  mapping  relative  to  the 
key. 

Simple  Reaction  Time 

Results  of  this  test  are  used  as  an  index  of  visuo-motor  response  timing. 

This  is  a  repeat  of  the  Simple  Reaction  Time  test  presented  earlier  in  the  battery.  This  test 
measures  simple  reaction  time  by  presenting  the  user  with  a  series  of  symbols  on  the  display. 
The  user  is  instructed  to  respond  as  quickly  as  possible  by  pressing  a  button  each  time  the 
stimulus  appears. 

Adapted  from:  C-SHOP  (2007).  ANAM4  TBI:  User  Manual.  Center  for  the  Study  of  Human 
Operator  Performance,  University  of  Oklahoma,  Norman,  OK.  Pages  15-18. 50 
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Appendix  D:  Breakdown  of  Total  ANAM4  Test  Results  by 
Year  and  Reason 


Through  23-JUL-15 

Pre-D 

Post-D 

Clin  Eval 

Injury 

Recovery 

Unspecified 

Percentage 

93.74% 

3.01% 

1.29% 

0.26% 

0.03% 

1.67% 

Sum  of  Count 

Column  Labels 

Period/Service 

Pre-D 

Post-D 

Clinical 

Injury 

Recovery 

Unspecified 

Total 

2007  Q2 

0 

0 

0 

0 

0 

335 

335 

Army 

0 

0 

0 

0 

0 

335 

335 

2007  Q3 

12,589 

27 

20 

1 

2 

242 

12,881 

Air  Force 

2 

0 

0 

1 

0 

2 

5 

Army 

12,538 

27 

18 

0 

2 

239 

12,824 

Non  Military 

49 

0 

2 

0 

0 

1 

52 

2007  Q4 

22,611 

3,853 

118 

9 

7 

3,713 

30,311 

Air  Force 

268 

148 

1 

0 

0 

6 

423 

Army 

22,324 

3,705 

116 

9 

7 

3,701 

29,862 

Non  Military 

19 

0 

1 

0 

0 

6 

26 

2008  Q1 

10,660 

93 

169 

4 

0 

116 

11,042 

Air  Force 

1,246 

2 

155 

1 

0 

31 

1,435 

Army 

9,370 

91 

14 

3 

0 

83 

9,561 

Marine  Corps 

4 

0 

0 

0 

0 

0 

4 

Navy 

29 

0 

0 

0 

0 

0 

29 

Non  Military 

11 

0 

0 

0 

0 

2 

13 

2008  Q2 

7,127 

1,228 

316 

5 

4 

314 

8,994 

Air  Force 

654 

5 

199 

0 

0 

8 

866 

Army 

6,461 

1,223 

116 

5 

4 

306 

8,115 

Marine  Corps 

1 

0 

0 

0 

0 

0 

1 

Non  Military 

11 

0 

1 

0 

0 

0 

12 

2008  Q3 

30,800 

736 

214 

20 

2 

1,051 

32,833 

Air  Force 

1,780 

60 

13 

2 

0 

53 

1,908 

Army 

25,331 

666 

173 

17 

2 

858 

27,057 

Coast  Guard 

0 

0 

1 

0 

0 

0 

1 

Marine  Corps 

3,237 

9 

25 

1 

0 

115 

3,387 

Navy 

390 

1 

1 

0 

0 

17 

409 

Non  Military 

62 

0 

1 

0 

0 

8 

71 

2008  Q4 

50,341 

10,996 

446 

104 

10 

1,489 

63,387 

Air  Force 

2,626 

247 

23 

7 

0 

55 

2,958 

Army 

45,936 

10,743 

403 

94 

9 

1,358 

58,544 

Marine  Corps 

1,497 

4 

8 

3 

1 

69 

1,582 

Navy 

198 

0 

7 

0 

0 

3 

208 

Non  Military 

84 

2 

5 

0 

0 

4 

95 

2009  Q1 

70,057 

4,161 

569 

22 

9 

2,042 

76,861 

Air  Force 

5,931 

47 

31 

0 

1 

141 

6,151 

Army 

37,410 

3,939 

342 

15 

4 

1,022 

42,733 

Coast  Guard 

4 

0 

1 

0 

0 

0 

5 

Marine  Corps 

22,377 

134 

184 

7 

4 

742 

23,448 

Navy 

4,242 

39 

10 

0 

0 

131 

4,422 

Non  Military 

93 

2 

1 

0 

0 

6 

102 

2009  Q2 

78,888 

2,707 

278 

14 

4 

1,622 

83,513 

Air  Force 

6,759 

37 

26 

0 

3 

68 

6,893 

Army 

58,381 

2,614 

203 

9 

1 

1,321 

62,529 

Coast  Guard 

143 

0 

0 

0 

0 

0 

143 

Marine  Corps 

9,766 

43 

44 

5 

0 

153 

10,011 

Navy 

3,757 

11 

5 

0 

0 

75 

3,848 

Non  Military 

82 

2 

0 

0 

0 

5 

89 

2009  Q3 

69,605 

1,477 

311 

22 

22 

1,076 

72,513 

Air  Force 

5,263 

10 

3 

0 

0 

42 

5,318 

Army 

50,159 

1,433 

271 

19 

21 

816 

52,719 

Coast  Guard 

35 

0 

0 

0 

0 

0 

35 

Marine  Corps 

11,324 

29 

37 

3 

1 

183 

11,577 

Navy 

2,745 

5 

0 

0 

0 

35 

2,785 

Non  Military 

79 

0 

0 

0 

0 

0 

79 

Adapted  from  AN  AM  Program  Office,  2015. 5 
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Period/Service 

Pre-D 

Post-D 

Clinical 

Injury 

Recovery 

Unspecified 

Total 

2009  Q4 

64,196 

2,886 

484 

12 

1 

1,133 

68,712 

Air  Force 

13,988 

134 

71 

1 

0 

179 

14,373 

Army 

41,511 

2,714 

295 

11 

1 

832 

45,364 

Coast  Guard 

146 

0 

0 

0 

0 

0 

146 

Marine  Corps 

5,578 

29 

109 

0 

0 

57 

5,773 

Navy 

2,811 

8 

8 

0 

0 

59 

2,886 

Non  Military 

162 

1 

1 

0 

0 

6 

170 

2010  Q1 

79,000 

479 

583 

25 

7 

1,537 

81,631 

Air  Force 

16,008 

85 

47 

4 

0 

295 

16,439 

Army 

43,480 

349 

328 

16 

5 

876 

45,054 

Coast  Guard 

2 

0 

1 

0 

0 

0 

3 

Marine  Corps 

15,472 

35 

186 

3 

2 

324 

16,022 

Navy 

3,845 

9 

19 

2 

0 

37 

3,912 

Non  Military 

193 

1 

2 

0 

0 

5 

201 

2010  Q2 

75,794 

838 

791 

73 

74 

1,367 

78,937 

Air  Force 

18,580 

53 

95 

6 

1 

394 

19,129 

Army 

41,423 

747 

581 

22 

5 

753 

43,531 

Coast  Guard 

200 

1 

0 

0 

0 

2 

203 

Marine  Corps 

11,775 

30 

98 

40 

68 

135 

12,146 

Navy 

3,647 

6 

13 

4 

0 

76 

3,746 

Non  Military 

169 

1 

4 

1 

0 

7 

182 

2010  Q3 

75,088 

565 

877 

629 

144 

1,033 

78,336 

Air  Force 

16,170 

45 

89 

16 

2 

236 

16,558 

Army 

45,745 

489 

654 

160 

17 

682 

47,747 

Coast  Guard 

102 

0 

0 

0 

0 

1 

103 

Marine  Corps 

8,882 

21 

118 

429 

114 

80 

9,644 

Navy 

3,788 

9 

14 

21 

11 

24 

3,867 

Non  Military 

401 

1 

2 

3 

0 

10 

417 

2010  <24 

68,292 

795 

894 

490 

37 

819 

71,327 

Air  Force 

13,975 

53 

35 

5 

2 

208 

14,278 

Army 

37,746 

693 

675 

132 

8 

473 

39,727 

Coast  Guard 

21 

0 

0 

0 

0 

0 

21 

Marine  Corps 

10,882 

15 

164 

334 

24 

65 

11,484 

Navy 

4,449 

27 

16 

13 

3 

36 

4,544 

Non  Military 

1,219 

7 

4 

6 

0 

37 

1,273 

2011  <21 

93,010 

4,948 

693 

499 

33 

1,461 

100,644 

Air  Force 

15,437 

53 

17 

8 

1 

321 

15,837 

Army 

54,161 

4,729 

384 

168 

8 

856 

60,306 

Coast  Guard 

254 

0 

0 

0 

0 

1 

255 

Marine  Corps 

15,875 

119 

253 

304 

24 

145 

16,720 

Navy 

5,241 

37 

28 

15 

0 

62 

5,383 

Non  Military 

2,042 

10 

11 

4 

0 

76 

2,143 

2011  Q2 

71,386 

6,190 

776 

483 

46 

851 

79,732 

Air  Force 

13,617 

51 

25 

5 

1 

221 

13,920 

Army 

41,057 

6,040 

371 

160 

13 

473 

48,114 

Coast  Guard 

103 

0 

0 

0 

0 

0 

103 

Marine  Corps 

10,174 

64 

343 

305 

31 

85 

11,002 

Navy 

4,953 

29 

33 

11 

1 

42 

5,069 

Non  Military 

1,482 

6 

4 

2 

0 

30 

1,524 

2011  Q3 

67,097 

3,535 

1,400 

331 

61 

910 

73,334 

Air  Force 

13,965 

42 

16 

9 

1 

238 

14,271 

Army 

37,321 

3,426 

910 

90 

38 

466 

42,251 

Coast  Guard 

105 

0 

0 

0 

0 

0 

105 

Marine  Corps 

9,609 

33 

425 

225 

19 

95 

10,406 

Navy 

3,496 

24 

37 

7 

3 

43 

3,610 

Non  Military 

2,601 

10 

12 

0 

0 

68 

2,691 
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Period/Service 

Pre-D 

Post-D 

Clinical 

Injury 

Recovery 

Unspecified 

Total 

2011  Q4 

54,558 

1,779 

869 

316 

24 

680 

58,226 

Air  Force 

11,239 

48 

15 

14 

0 

222 

11,538 

Army 

27,589 

1,626 

584 

157 

9 

280 

30,245 

Coast  Guard 

112 

0 

0 

0 

0 

0 

112 

Marine  Corps 

9,887 

27 

244 

134 

13 

65 

10,370 

Navy 

2,721 

68 

22 

9 

2 

32 

2,854 

Non  Military 

3,010 

10 

4 

2 

0 

81 

3,107 

2012  Q1 

63,447 

2,674 

2,352 

187 

27 

860 

69,547 

Air  Force 

12,349 

45 

14 

8 

1 

114 

12,531 

Army 

33,224 

2,525 

670 

101 

10 

455 

36,985 

Coast  Guard 

135 

0 

0 

0 

0 

0 

135 

Marine  Corps 

10,066 

47 

1,524 

62 

16 

83 

11,798 

Navy 

4,236 

39 

141 

15 

0 

54 

4,485 

Non  Military 

3,437 

18 

3 

1 

0 

154 

3,613 

2012  Q2 

57,535 

880 

1,817 

333 

12 

853 

61,430 

Air  Force 

11,082 

33 

13 

20 

0 

117 

11,265 

Army 

31,082 

767 

716 

240 

11 

484 

33,300 

Coast  Guard 

139 

0 

0 

0 

0 

1 

140 

Marine  Corps 

9,020 

22 

1,035 

62 

1 

50 

10,190 

Navy 

2,965 

39 

42 

8 

0 

11 

3,065 

Non  Military 

3,247 

19 

11 

3 

0 

190 

3,470 

2012  Q3 

62,685 

860 

1,081 

435 

15 

855 

65,931 

Air  Force 

11,547 

28 

23 

15 

0 

113 

11,726 

Army 

39,023 

763 

375 

369 

13 

527 

41,070 

Coast  Guard 

87 

0 

1 

0 

0 

1 

89 

Marine  Corps 

5,860 

10 

639 

45 

2 

12 

6,568 

Navy 

2,960 

40 

32 

5 

0 

10 

3,047 

Non  Military 

3,208 

19 

11 

1 

0 

192 

3,431 

2012  <24 

62,948 

532 

1,425 

200 

9 

985 

66,099 

Air  Force 

8,872 

43 

6 

7 

0 

120 

9,048 

Army 

40,147 

419 

372 

124 

4 

643 

41,709 

Coast  Guard 

17 

0 

0 

0 

0 

0 

17 

Marine  Corps 

8,148 

20 

975 

59 

5 

16 

9,223 

Navy 

2,314 

24 

66 

10 

0 

17 

2,431 

Non  Military 

3,450 

26 

6 

0 

0 

189 

3,671 

2013  <21 

62,132 

1,095 

1,228 

87 

15 

972 

65,529 

Air  Force 

12,721 

52 

187 

11 

0 

138 

13,109 

Army 

36,658 

985 

426 

60 

10 

587 

38,726 

Coast  Guard 

114 

0 

0 

0 

0 

0 

114 

Marine  Corps 

6,587 

20 

575 

13 

4 

11 

7,210 

Navy 

2,481 

15 

32 

3 

0 

24 

2,555 

Non  Military 

3,571 

23 

8 

0 

1 

212 

3,815 

2013  Q2 

58,119 

284 

558 

60 

13 

706 

59,740 

Air  Force 

10,335 

33 

97 

9 

0 

116 

10,590 

Army 

34,167 

194 

405 

41 

11 

420 

35,238 

Coast  Guard 

75 

0 

0 

0 

0 

0 

75 

Marine  Corps 

8,902 

14 

38 

7 

1 

43 

9,005 

Navy 

2,175 

33 

14 

3 

1 

15 

2,241 

Non  Military 

2,465 

10 

4 

0 

0 

112 

2,591 

2013  Q3 

52,872 

217 

1,372 

132 

6 

652 

55,251 

Air  Force 

11,040 

34 

17 

14 

0 

120 

11,225 

Army 

29,561 

153 

259 

111 

6 

310 

30,400 

Coast  Guard 

69 

0 

0 

0 

0 

0 

69 

Marine  Corps 

7,397 

12 

961 

2 

0 

22 

8,394 

Navy 

2,187 

7 

131 

5 

0 

22 

2,352 

Non  Military 

2,618 

11 

4 

0 

0 

178 

2,811 
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Period/Service 

Pre-D 

Post-D 

Clinical 

Injury 

Recovery 

Unspecified 

Total 

2013  Q4 

47,433 

221 

942 

47 

8 

587 

49,238 

Air  Force 

9,772 

41 

12 

17 

0 

156 

9,998 

Army 

26,444 

139 

217 

29 

8 

302 

27,139 

Coast  Guard 

37 

0 

0 

0 

0 

1 

38 

Marine  Corps 

6,768 

8 

671 

0 

0 

5 

7,452 

Navy 

1,997 

19 

35 

0 

0 

12 

2,063 

Non  Military 

2,415 

14 

7 

1 

0 

111 

2,548 

2014  Q1 

48,216 

237 

1,015 

9 

5 

458 

49,940 

Air  Force 

11,103 

36 

31 

2 

2 

139 

11,313 

Army 

25,023 

153 

224 

7 

2 

165 

25,574 

Coast  Guard 

102 

0 

0 

0 

0 

0 

102 

Marine  Corps 

7,532 

26 

676 

0 

1 

9 

8,244 

Navy 

2,007 

8 

82 

0 

0 

17 

2,114 

Non  Military 

2,449 

14 

2 

0 

0 

128 

2,593 

2014  Q2 

50,585 

482 

670 

30 

3 

522 

52,292 

Air  Force 

9,957 

50 

20 

6 

0 

170 

10,203 

Army 

27,569 

380 

281 

20 

2 

244 

28,496 

Coast  Guard 

76 

0 

0 

0 

0 

0 

76 

Marine  Corps 

7,843 

28 

341 

3 

0 

6 

8,221 

Navy 

2,945 

10 

26 

1 

1 

12 

2,995 

Non  Military 

2,195 

14 

2 

0 

0 

90 

2,301 

2014  Q3 

43,617 

142 

657 

37 

6 

439 

44,898 

Air  Force 

11,575 

39 

67 

5 

1 

182 

11,869 

Army 

23,102 

89 

381 

29 

5 

218 

23,824 

Coast  Guard 

86 

0 

0 

0 

0 

0 

86 

Marine  Corps 

6,139 

8 

195 

2 

0 

9 

6,353 

Navy 

1,752 

2 

12 

1 

0 

11 

1,778 

Non  Military 

963 

4 

2 

0 

0 

19 

988 

2014  Q4 

38,908 

225 

357 

76 

2 

454 

40,022 

Air  Force 

9,129 

45 

51 

6 

1 

238 

9,470 

Army 

23,307 

157 

264 

67 

1 

187 

23,983 

Coast  Guard 

10 

0 

0 

0 

0 

0 

10 

Marine  Corps 

3,559 

7 

36 

2 

0 

1 

3,605 

Navy 

2,303 

6 

2 

1 

0 

17 

2,329 

Non  Military 

600 

10 

4 

0 

0 

11 

625 

2015  Q1 

43,125 

131 

316 

18 

4 

377 

43,971 

Air  Force 

9,413 

27 

9 

3 

0 

80 

9,532 

Army 

23,088 

92 

298 

12 

4 

275 

23,769 

Coast  Guard 

12 

0 

0 

0 

0 

0 

12 

Marine  Corps 

8,326 

6 

5 

3 

0 

4 

8,344 

Navy 

1,707 

3 

3 

0 

0 

7 

1,720 

Non  Military 

579 

3 

1 

0 

0 

11 

594 

2015  Q2 

33,481 

134 

186 

11 

0 

240 

34,052 

Air  Force 

7,335 

20 

7 

5 

0 

41 

7,408 

Army 

19,103 

104 

164 

4 

0 

176 

19,551 

Coast  Guard 

9 

0 

0 

0 

0 

0 

9 

Marine  Corps 

4,563 

4 

12 

1 

0 

3 

4,583 

Navy 

1,888 

4 

3 

1 

0 

10 

1,906 

Non  Military 

583 

2 

0 

0 

0 

10 

595 

Grand  Total 

1,726,202 

55,407 

23,784 

4,721 

612 

30,751 

1,841,489 
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Appendix  E:  Proposed  Research  Highlighted  in  Literature 


Area  of  Interest 

Recommendation 

Source 

If  multidimensional  clinical  assessment  was 
used  following  an  mTBI,  does  the  addition 
of  baseline  pre-injury  test  scores  contribute 
in  a  meaningful  way  to  medical  management 
and  return-to-duty  decisions? 

Dr.  Grant  Iverson 

Impact  of  baselines  on 
post-injury  mTBI 
management 

Assess  the  reliability  and  validity  of  a 
baseline  versus  no  baseline  model  of  CNT 
assessment  when  measuring  the  acute  effects 
of  sports  related  concussion. 

Resch  et  al  201374 

Conduct  programmatic  research  relating  to 
the  strengths  and  limitations  of  baseline 
testing  for  improving  the  accuracy  of 
neuropsychological  assessment  and 
determining  whether  improved  accuracy 
contributes  to  improved  management  of  this 
injury  in  athletes. 

Iverson  and  Schatz,  20 15 16 

What  is  the  best  methodology  for 
interpreting  post-concussion  test  scores  when 
baseline  (i.e.,  pre -deployment)  test  scores  are 
available? 

Dr.  Grant  Iverson 

What  are  the  best  clinical  algorithms  for 
interpreting  ANAM  scores  following  a  mild 
traumatic  brain  injury  in  the  absence  of 
baseline  pre-injury  test  scores? 

Dr.  Grant  Iverson 

Is  having  pre-deployment  ANAM  scores 
superior  to  not  having  these  scores  when 
interpreting  scores  following  a  mild 
traumatic  brain  injury? 

Dr.  Grant  Iverson87 

ANAM  Score 

Is  the  reliable  change  methodology  or 
regression-based  change  models  preferred 
for  interpreting  retest  scores  on  the  ANAM? 

Dr.  Grant  Iverson 

Interpretation 

Develop  and  evaluate  more  sophisticated 
methods  for  interpreting  change  on  cognitive 
testing  (e.g.,  refinements  of  the  reliable 
change  methodology,  such  as  correction  for 
practice  and  stratification  of  confidence 
intervals  for  change  based  on  level  of 
baseline  performance;  and  use  of 
standardized  regression  models).  Apply 
multivariate  base  rate  analyses  to  the  reliable 
change  methodology  to  quantify  the 
likelihood  of  showing  one  or  more  reliable 
change  scores  when  multiple  change  scores 
are  considered  simultaneously. 

Iverson  and  Schatz,  20 15 16 

Develop  and  evaluate  clinical  algorithms, 
with  known  false  positive  rates,  for 

Iverson  and  Schatz,  20 15 16 
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Area  of  Interest 

Recommendation 

Source 

identifying  and  quantifying  cognitive 
impairment  following  concussion. 

ANAM  Test/Sub-Test 
Sensitivity 

Is  the  ANAM  sensitive  to  cognitive 
difficulties  associated  with  depression,  pain, 
acute  traumatic  stress,  or  some  combination 
of  these  factors? 

Dr.  Grant  Iverson 

Which  ANAM  subtests  are  most  sensitive  to 
cognitive  deficits  in  those  with  moderate  or 
severe  TBIs  after  1  year  of  recovery  time? 

Dr.  Grant  Iverson 

Impact  of 

Devices/Technology  on 
ANAM  Scores 

What  are  the  psychometric  implications  of 
completing  the  ANAM  on  different  devices? 

Dr.  Grant  Iverson 

Test 

Setting/Administration 

What  are  the  psychometric  implications  of 
completing  the  ANAM  in  individual  or 
group  settings? 

Dr.  Grant  Iverson 

Examine  how  confusion  or  misunderstanding 
on  the  part  of  the  subject,  regarding  the  test 
instructions  or  procedures,  influences  the 
probability  of  being  flagged  by  a  validity 
indicator. 

Iverson  and  Schatz,  20 15 16 

Establish  guidelines  for  “best  practices”  in 
the  administration  of  CNTs  across  multiple 
sports,  age  groups  and  levels  of  competition. 

Resch  et  al  201324 

Comparison  of  ANAM 
to  Other  Tests 

Does  the  ANAM  have  comparable  or 
superior  reliability  and  clinical  usefulness  as 
other  brief  computerized  cognitive 
assessment  batteries? 

Dr.  Grant  Iverson 

Further  demonstrate  and  report  the 
psychometric  properties  of  each  CNTs  in  the 
assessment  of  sports  related  concussion 
through  prospective  studies  of  adult  and 
youth  athletes. 

Resch  et  al  201324 

Investigate  the  psychometric  evidence  for 
equivalence  of  alternate  forms  provided  by 
each  CNT. 

Resch  et  al  201324 

Timing  of  testing 

Investigate  the  added  value  of  CNTs 
compared  to  traditional  neuropsychological 
tests  as  measures  of  cognitive  recovery  at 
sequential  time  points  following  injury. 

Resch  et  al  201324 

Determine  if  cognitive  functioning,  assessed 
in  the  first  72  hours  post  injury,  has 
prognostic  value  for  predicting  typical  versus 
slow  recovery. 

Iverson  and  Schatz  20 15 16 

Pursue  analogue  malingering  studies  to 

Iverson  and  Schatz  20 15 16 
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Area  of  Interest 

Recommendation 

Source 

better  understand  how  people  under-perform 
on  computerized  testing. 

Test/Re-test  Effects 

Evaluate  and  improve,  if  possible,  the  test- 
re -test  reliability  of  traditional  and 
computerized  cognitive  tests.  Determine  if 
there  are  differences  in  the  magnitude  of 
practice  effects  based  on  the  domain  of 
cognitive  functioning  assessed  and  whether 
the  test  is  paper-pencil  or  computerized. 

Iverson  and  Schatz  20 15 16 

Sources  of  Error 

Investigate  additional  sources  of  random  and 
systematic  error  that  may  influence  CNT 
performance. 

Resch  et  al  201324 

Blast  Injuries  and 
Multiple  Concussions 

Continued  study  of  combat  related  blast 
concussions  is  important  to  our  efforts  to 
quickly  and  safely  return  service  members  to 
duty,  treat  persistent  concussion  symptoms, 
and  avoid  long-term  sequel  and  CTE. 

Kelly  et  al  2012bU 

[There  are]  [ljongitudinal  bodies  of  data 
currently  available  from  which  to  draw 
conclusions  regarding  the  impact  of  multiple 
concussions  during  combat. 

Kelly  et  al  2012bU 
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Appendix  F:  Meetings  and  Briefings 


October  2,  2014 

On  this  teleconference  members  discussed  the  tasking  and  a  potential  way  forward.  There  were 
no  briefings  at  this  meeting. 

December  18,  2014 

On  this  teleconference  members  discussed  the  tasking,  the  Automated  Neuropsychological 
Assessment  Metrics  (ANAM)  program,  and  ANAM  data  analyses  with  subject  matter  experts. 

Subject  matter  experts  in  attendance: 

•  Ms.  Kathy  Helmick,  Deputy  Director,  Defense  and  Veterans’  Brain  Injury  Center 

•  COL  Sidney  Hinds,  Director,  Defense  and  Veterans’  Brain  Injury  Center 

•  Dr.  Donald  Marion,  Senior  Clinical  Consultant,  Clinical  Affairs,  Defense  and  Veterans’ 

Brain  Injury  Center 

•  Dr.  Mark  Kelly,  Program  Director,  Postdoctoral  Fellowship  in  Clinical  Neuropsychology, 
Walter  Reed  National  Military  Medical  Center 

February  17, 2015 

On  this  teleconference  members  discussed  the  way  ahead  and  reviewed  their  Terms  of  Reference 
and  Guiding  Principles.  There  were  no  briefings  at  this  meeting. 

March  11,  2015 

Falls  Church,  Virginia 

Members  met  with  subject  matter  experts  to  discuss  the  ANAM4  database  and  testing  as  well  as 
evaluation,  treatment,  and  management  of  mTBI.  Members  also  reviewed  their  Terms  of 
Reference  and  Guiding  Principles,  and  discussed  the  way  ahead. 

Subject  matters  in  attendance: 

•  Dr.  John  Meyers,  Neuropsychologist,  ANAM  Program  Director/Chief  Neurocognitive 
Assessment  Branch,  Rehabilitation  and  Reintegration  Division;  Headquarters,  Department  of 
the  Army  (HQDA),  Office  of  the  Army  Surgeon  General 

•  MAJ  Robert  Parish,  Medical  Service  Corps  Deputy  Chief,  Department  of  Behavioral  Health 
Officer  in  Charge,  mild  Traumatic  Brain  Injury  (mTBI)  Clinic  Clinical  Neuropsychologist 

•  LCDR  Steven  Porter,  Clinical  Neuropsychologist,  Midshipmen  Development  Center,  United 
States  Naval  Academy 

•  Dr.  Michael  Russell,  Director,  Veterans  Integrated  Service  Network  17  Center  of  Excellence 
for  Research  on  Returning  War  Veterans 

•  CAPT  Jack  Tsao,  Director  of  Traumatic  Brain  Injury  Programs  for  the  U.S.  Navy  Bureau  of 
Medicine  and  Surgery;  Professor  of  Neurology,  Uniformed  Services  University  of  the  Health 
Sciences;  Fellow  of  the  American  Academy  of  Neurology 
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April  27,  2015 

On  this  teleconference  members  discussed  issues  in  neurocognitive  assessment  of  mTBI  with 
subject  matter  experts.  Members  also  reviewed  the  draft  report  and  future  briefings  needed. 

Subject  matter  experts  in  attendance: 

•  Dr.  Grant  Iverson,  Director,  Sports  Concussion  Program,  Massachusetts  General  Hospital  for 
Children;  Director,  Neuropsychology  Outcome  Assessment  Laboratory,  Department  of 
Physical  Medicine  and  Rehabilitation,  Harvard  Medical  School 

•  Dr.  Philip  Schatz,  Director  Behavioral  Neurosciences  Program,  Saint  Joseph’s  University 

May  26,  2015 

On  this  teleconference  members  discussed  issues  in  neurocognitive  assessment  of  mTBI  with 
subject  matter  experts.  Members  also  reviewed  the  draft  report  outline. 

Subject  matter  experts  in  attendance: 

•  Dr.  Alison  Cemich,  Director,  National  Center  for  Medical  Rehabilitation  Research  at  the 
National  Institute  of  Child  Health  and  Human  Development 

•  Dr.  Michael  McCrea,  Professor  of  Neurosurgery  and  Neurology;  Director  of  Brain  Injury 
Research,  Medical  College  of  Wisconsin 

July  9,  2015 

On  this  teleconference  members  reviewed  and  edited  the  draft  outline  and  report.  There  were  no 
briefings  at  this  meeting. 

July  23,  2015 

On  this  teleconference  members  reviewed  and  edited  the  draft  report.  There  were  no  briefings  at 
this  meeting. 

August  31,  2015 

On  this  teleconference  members  discussed  computerized  neurocognitive  testing  with  a  subject 
matter  expert.  Members  also  reviewed  and  edited  the  draft  report. 

Subject  matter  expert  in  attendance: 

•  Dr.  Wayne  Chappelle,  Consultant  to  the  Surgeon  General,  Aeromedical  Clinical 
Psychology,  U.S.  Air  Force  School  of  Aerospace  Medicine 

September  28,  2015 

On  this  teleconference  members  discussed  the  ANAM4  database  and  baseline  testing  with 
subject  matter  experts.  Members  also  reviewed  and  edited  the  draft  report. 
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Subject  matter  experts  in  attendance: 

•  Dr.  John  Meyers,  Neuropsychologist,  ANAM  Program  Director/Chief  Neurocognitive 
Assessment  Branch,  Rehabilitation  and  Reintegration  Division,  HQDA,  Office  of  the 
Army  Surgeon  General 

•  Dr.  Stephanie  Maxfield-Panker,  Acting  TBI  Program  Director,  Office  of  the  Army 
Surgeon  General,  Rehabilitation  and  Reintegration  Division 

October  19,  2015 

On  this  teleconference  members  discussed  the  quality  of  computerized  neurocognitive 
assessments  with  subject  matter  experts.  Members  also  reviewed  and  edited  the  draft  report. 

Subject  matter  experts  in  attendance: 

•  Dr.  David  Cifu,  National  Director  of  Physical  Medicine  &Rehabilitation  (PM&R) 
Program  Office,  Veterans  Health  Administration;  Chairman  and  Herman  J.  Flax,  M.D. 
Professor  Department  of  PM&R,  Virginia  Commonwealth  University 

•  Dr.  Chris  Giza,  Professor  of  Pediatric  Neurology  and  Neurosurgery,  UCLA  Brain  Injury 
Research  Center 

October  26,  2015 

On  this  teleconference  members  reviewed  and  edited  the  draft  findings  and  recommendations. 
No  subject  matter  experts  attended  this  teleconference. 

November  9,  2015 

Defense  Health  Board  Meeting 

MacDill  Air  Force  Base,  Florida 

Dr.  David  Hovda,  Subcommittee  chair,  presented  the  deliberative  pre-decisional  draft  of  the 
report. 
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Appendix  G:  Acronyms 


AMEDD 

United  States  Army  Medical  Department 

ANAM4 

Automated  Neuropsychological  Assessment  Metrics  4 

CNAD 

Computerized  Neurocognitive  Assessment  Device 

CNT 

Computerized  Neurocognitive  Test 

C-SHOP 

Center  for  the  Study  of  Human  Operator  Performance 

DANA 

Defense  Automated  Neurobehavioral  Assessment 

DCoE 

Defense  Centers  of  Excellence  for  Psychological  Health  and  Traumatic  Brain 
Injury 

DHB 

Defense  Health  Board 

DoD 

Department  of  Defense 

DoDI 

Department  of  Defense  Instruction 

DVBIC 

Defense  and  Veterans’  Brain  Injury  Center 

ImPACT 

Immediate  Post-Concussion  Assessment  and  Cognitive  Testing 

MACE 

Military  Acute  Concussion  Evaluation 

mTBI 

Mild  Traumatic  Brain  Injury 

NCAT 

Neurocognitive  Assessment  Tool 

NDAA 

National  Defense  Authorization  Act 

OEF 

Operation  Enduring  Freedom 

OIF 

Operation  Iraqi  Freedom 

PRT 

Procedural  Reaction  Time 

PTSD 

Post-Traumatic  Stress  Disorder 

SRT 

Simple  Reaction  Time 

TBI 

Traumatic  Brain  Injury 

USASOC 

United  States  Army  Special  Operations  Command 

USD(R&R) 

Under  Secretary  of  Defense,  Personnel  and  Readiness 

USNA 

United  States  Naval  Academy 
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